Abstract

Surveillance systems are important tools for law enforcement agencies for fighting crimes. Surveillance control rooms have two main duties: live monitoring the surveillance areas, and crime solving by investigating the archives. To support these difficult tasks, several significant solutions from the research and market fields have been proposed. However, the lack of generic and precise models for video content representation make the building of fully automated intelligent video analysis and description system a challenging task. Furthermore, the application domain still shows a big gap between the research field and the real practical needs, it also shows a lack between these real needs and the onmarket video analytics tools. Consequently, in conventional surveillance systems, live monitoring and investigating the archives still rely mostly on human operators. This thesis proposes a novel approach for textual describing important contents in videos surveillance scenes, based on new generic context-free "VSSD ontology", with focus on two objects interactions. The proposed ontology presents a new generic flexible and extensible ontology dedicated for video surveillance scenes description. While analysing and understanding variety of video scenes, our approach introduces many new concepts and methods concerning mediation and action at a distant, abstraction in the description, and a new manner of categorizing the scenes. It introduces a new heuristic way to discriminate between deformable and non-deformable objects in the scenes. It also highlights and exports important features for better video objects interactions learning classifications and for better description. These features, if used as key parameters in video analytics tools, are much suitable for supporting surveillance systems operators through generating alerts, and intelligent search. Moreover, our system outputs can support police incidents reports, according to investigators needs, with many types of automatic textual description based on new wellstructured rule-based schemas or templates.

Additionally, in this thesis, many important propositions were made, driven by practical experience, to reduce the existing gaps between the surveillance systems operators' needs from one side, the research field and the commercial (industry) field from the other side. These propositions encounter the research field, and the practical one, especially at the level of future intelligence video analytics development and integration with other systems. Some of these propositions are innovative yet simple to be applied, which can bring great benefits and optimize the use for surveillance systems operators when live monitoring, investigating, and analysing the crimes.

Résumé

Les systèmes de vidéosurveillance sont des outils importants pour les agences chargées de l'application de la loi dans la lutte contre la criminalité. Les chambres de contrôle de la vidéosurveillance ont deux fonctions principales : surveiller en direct les zones de surveillance et résoudre les infractions en enquêtant les archives. Pour soutenir ces tâches difficiles, plusieurs solutions significatives issues des domaines de la recherche et du marché ont été proposées. Cependant, le manque de modèles génériques et précis pour la représentation du contenu vidéo fait de la construction d'un système intelligent et automatisé capable d'analyser et de décrire des vidéos une tâche ardue. De plus, le domaine d'application montre toujours un écart important entre le domaine de la recherche et les besoins réels, ainsi qu'un manque entre ces besoins réels et les outils d'analyse vidéo dans le marché. Par conséquence, jusqu'à présent dans les systèmes de surveillance conventionnels, la surveillance en direct et la recherche dans des archives reposent principalement sur des opérateurs humains.

Cette thèse propose une nouvelle approche pour la description textuelle de contenus importants dans des scènes de vidéosurveillance, basée sur une nouvelle «ontologie VSSD» générique, sans contexte, centrée sur les interactions entre deux objets. L'ontologie proposée est générique, flexible et extensible, dédiée à la description de scènes de vidéosurveillance. Tout en analysant les différentes scènes vidéo, notre approche introduit de nombreux nouveaux concepts et méthodes concernant la médiation et l'action distante, la description synthétique, ainsi qu'une nouvelle façon de segmenter la vidéo et de classer les scènes. Nous introduisons une nouvelle méthode heuristique de distinction entre les objets déformables et non déformables dans les scènes. Nous proposons également des caractéristiques importantes pour une meilleure classification des interactions entre les objets vidéo, basée sur l'apprentissage, et une meilleure description. Ces caractéristiques, si elles sont utilisées comme paramètres clés dans les outils d'analyse vidéo, sont bien adaptées pour aider les opérateurs de systèmes de surveillance à travers des générations d'alertes, et une recherche intelligente.

De plus, nos sorties système peuvent prendre en charge les rapports d'incidents de police, selon les besoins des enquêteurs, avec de nombreux types de descriptions textuelles automatiques basées sur de nouveaux schémas ou modèles, bien structurés et basés sur des règles.

Enfin, dans cette thèse, de nombreuses propositions importantes ont été faites, s'appuyant sur l'expérience pratique, pour réduire les écarts existants entre les besoins des opérateurs de systèmes de surveillance d'un côté, le domaine de la recherche et le domaine commercial (de l'industrie) de l'autre côté. Ces propositions engagent le domaine de la recherche et le domaine pratique, en particulier au niveau du développement futur des produits intelligents d'analyse de vidéos et de l'intégration avec d'autres systèmes. Certaines de ces propositions sont novatrices mais simples à appliquer, ce qui peut apporter d'importants avantages et optimiser l'utilisation par les opérateurs de systèmes de surveillance lors du suivi en direct, de l'enquête et de l'analyse des crimes.

Table of Contents

LIST OF TABLES

LIST OF FIGURES

Figure I-1: An example of video captioning architecture using sequence learning approach, taken from (Z. [START_REF] Wu | Deep Learning for Video Classification and Captioning[END_REF] -20: Interaction existence classification for the scene "LeftBox" after filtering using [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF], shows an interaction began to appear at window starting at frame 40......

I. Introduction

In an era of rapid technological development on many fields (social, political, economic, security, …), and the accompanying changes of the crime aspects and its methods, getting more sophisticated by the day, the law enforcement agencies duties towards their citizens are becoming more and more difficult to enforce. These duties revolve around the protection of persons, property and freedom, the maintaining of order and the strengthening of security, public safety and the application of laws and regulations. Thus, law enforcement agencies, are in continuous search for effective public safety and security strategies to help deal with criminal and terrorist acts.

The new face of dealing and fighting with crimes is through collecting data, and transforming this data into intelligence. Among the most modern public safety and law enforcement tools is the surveillance system, where the video surveillance is a big source of data and the strong point of the most investigations.

The rapid progress in technology, Multiplexing, digital technology, NVRs, storage and processing made enormous progress in surveillance systems. For that, the deployment of video surveillance systems worldwide has grown exponentially in recent years. Many of large cities have concerns about crimes, terrorist attacks, incidents and antisocial behaviour problems, such as fights, vandalism, breaking and accidents, often these cities have video cameras already installed in the streets and around the important sites. Visual surveillance is now used to monitor the security of sensitive areas, as a risk management and crime reduction tool, such as in public places, schools, banks, shopping malls, transport infrastructures (e.g. airports, underground stations), hospitals, government buildings and borders.

One of the most important duties and goals of surveillance systems is to live monitoring the surveillance areas, in case incident occurs, actions should be taken. Another main focus is crime solving by investigating the archives. Two tasks are difficult to achieve, due to lack of human resources for active monitoring and of accurate parameters concerning archive indexation. Most, video surveillance recordings are indexed with rough descriptors such as time, camera ID and some photometric parameters.

Surveillance systems produce large amounts of video data which are stored for immediate or future use. Years of video surveillance are recorded. A crucial need is to make sense of this massive quantity of visual data. Surveillance videos are generous in motion information which rises up as one of the most important clue to identify the dynamic content of videos. Extraction and analysis of motion information in videos are essential in content-based surveillance video analysis and understanding. Detecting and understanding an incident is a simple mission for human, but it is very complicated for machine.

There is a fundamental need to extract automatically meaningful content and produce high-level scheme or descriptions of the activities. Such a system can help effectively to generate alerts to assist the live monitoring, and can help intelligently to index, organize, and retrieve valuable information from surveillance video databases, and finally to automatically generate useful reports.

Several significant solutions from the research and market fields have been proposed. However, the lack of generic and precise models for video content representation make building of fully automated intelligent video analysis and description a challenging task. This lack is due to the high complexity of video scenes, and its diversity from the context to the objects and actions types. Furthermore, for these reasons and many others, the application domain still shows a big gap between the research field and the real needs, also show a lack between these real needs and the on-market video analytics tools. Unfortunately, till now, both tasks of crime solving and live monitoring in conventional visual surveillance systems rely mostly on human operators; either to dig hard threw hundreds of hours in the archives, or to monitor actively hundreds of cameras. This thesis proposes a novel approach for textual description of the video scenes. We claim it to be a new approach for a general knowledge-based context-independent applicable in real-world surveillance video. Our approach is based on a proposed ontology, which combines objects features and derive high-level information; our ontology is methodologic and easily expendable.

From the perspective of an experienced Major, head of CCTV control room, the main two concerns of the approach were, to: first automatically extract useful information for investigating the archives and setting up alerts in real-cases, and second, to present it in an understandable way for the system operators by proposing new sentence representations. For that, well-structured schemas can be applied to generate incident scene descriptions similar to ones used in police reports. These schemas are also called templates.

"On Friday 17/05/2019 at 15:06:39: A person "2" moves, in intersection spot " 35.487536), on the left of "Hamra" street, heading immediately north, toward the person "1", occurring respectively irregularity in its shape, and big changes occurring respectively on its surface having now smaller one, and having respectively considerable decreasing of its Speed.

"The two objects are approaching; a distant aggressive interaction occurs between them."

Example of scene description and of object interaction description.

A description as the one in the above example can be very helpful for the surveillance system operators. It is based on many useful parameters such as objects characteristics, and inter-objects parameters. Those characteristics can be set up to generate alerts. They also can be queried for, in the archives. This thesis also highlights the existing gap between the research field and the surveillance system operators' needs from one side and the gap between the latter one with the commercial (industry) field. Many propositions were made driven by my personal experience as a researcher and a CCTV Control Room manager.

The research works presented in this thesis have been conducted under high constraints of applicability in a professional context. They took place in the framework of cooperation between IRIT Laboratory in the University of Paul Sabatier -Toulouse, France, and the Lebanese university, Beirut, Lebanon, and Beirut CCTV control room, Lebanon.

I.1. State of the art

Understanding and textually describing a video scene is an easy task for most humans, but is still a complex task to the computer. Automatic video scene description includes understanding and differentiating between the multiplicity of the backgrounds, the objects, the interactions, the scenes types, and the temporal order of incidents and events. Moreover, it requires a translation of the information into a comprehensible textual description or what is known as natural language.

The textual description, in general, can be used to improve wide range of applications like human-robot interaction, scene descriptor for blind, summary of (web-) videos, medical diagnosis, surveillance systems, robotics, military systems and others. In specific, for surveillance systems and traffic surveillance in cities, generating alerts for the observers and intelligently investigating the archives for the investigators.

In the last decades, researchers have studied multiple strategies and ontologies to bridge the gap existing between visual content and textual description. For that, Computer vision and Natural language Processing (NLP) fields addresses such a problem, separately and also, some workshops have been held on both areas [START_REF] Andrei | Language and Vision Workshop -2018[END_REF].

Being stretched, over more than two decades, and having so many applications have made this area of research very wide. Therefore, it's quite challenging to recapitulate and categorize all the works done in this area, especially as each contribution might differ according to the needs, outputs, methodologies, automation degree, used methods, and even sometimes trends.

In addition, researches in the related fields to video description like connecting words to pictures, image captioning, video to text, narrating images in natural language sentences, video captioning, video summarization, behaviour descriptions, natural-language video descriptions, and visual recognition and description, may share common methods or follow similar methodology.

Not restricted to video surveillance, two main approaches can be noticed: 1-Behaviour understanding and sentences generation [START_REF] Barbu | Video In Sentences Out[END_REF] [START_REF] Thomason | Integrating language and vision to generate natural language descriptions of videos in the wild[END_REF]) [START_REF] Guadarrama | Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition[END_REF], (A. Rohrbach et al., 2014), (R. [START_REF] Xu | Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework[END_REF]: this approach's name was chosen because most of the researches following this approach are mainly focusing on two stages: a. Behaviour understanding and content identification: also known as Knowledgebased or deterministic approaches. Extracting all needed features for identifying the semantic content and understanding the behaviour and the scene. A variety of approaches, techniques and methods have been proposed, we mention, object detection and segmentation, object classification, object recognition, multi-object tracking, trajectory analysis, action analysis, activity classification and recognition, and others. Typically, this part may involve training individual classifiers to identify background, objects, and actions in the scenes. As our main approach will focus on extracting many of the features, we will furthermore discuss the state of art of each of the corresponding methods in chapter IV.

b. Sentence generation: generating a sentence, normally, based on a template with syntactical structure (like Subject-Verb-Object SVO tuples, place, and scene). It may uses also a probabilistic model to map the most important visual content results from the video for each of the template categories to generate a sentence.

Some approaches from the literature are explained in the next section:

-In (S. [START_REF] Park | Event semantics in two-person interactions[END_REF], the authors present a method to describe two-person interactions in a semantic NL description. For that they detect body posture after integrating individual body parts (head, torso, arms and legs) recognized using Bayesian networks (BNs). To recognize specific interactions, they used decision tree with rule-based spatial and temporal constraints. Then they map it into verb phrases using sequential and simultaneous recognitions of the predefined interactions.

Human interactions are then represented as cause-effect semantics between syntactical <agent-motion-target> triplets. -In [START_REF] Farhadi | Every Picture Tells a Story: Generating Sentences from Images[END_REF], the authors proposed a system based on the detection of one object per image, then map it to the corresponding textual descriptions using a predefined language templates (triplet of S-V-O). -In [START_REF] Barbu | Video In Sentences Out[END_REF], the authors use a dynamic programming approach combined with Hidden Markov Models to obtain verb labels for short video clips, for producing sentential descriptions. -In [START_REF] Guadarrama | Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition[END_REF], the authors used semantic hierarchies to indicate the appropriate level of the accuracy and specificity of sentence fragments. -In (M. [START_REF] Rohrbach | Translating video content to natural language descriptions[END_REF], the authors incorporated semantic unaries and handcentric features and utilized a CRF-based approach to generate video descriptions. Their method is composed mainly of two steps; first to generate semantic representation models, they feed a Conditional random field CRF using dense trajectories and SIFT features and temporal context reasoning. Second they translate it to natural language using Statistical Machine Translation (SMT). -In (R. [START_REF] Xu | Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework[END_REF], for video sentence generation the authors designed a deep joint video-language embedding model. -In [START_REF] Hanckmann | Automated Textual Descriptions for a Wide Range of Video Events with 48 Human Actions[END_REF], the authors proposed a hybrid method to generate textual descriptions of video actions. Their system has mainly two parts, an action classifier and a description generator. They detect and classify 48 actions in a video using the Bag-of features. The description generator, a rule-based method, finds the actors (persons or objects) in the video and connects these to the appropriate verbs. -Krishnamoorthy et al. (Krishnamoorthy, Malkarnenkar, Mooney, Saenko, & Guadarrama, 2013) they were the first to introduce early works of describing open domain short videos data (YouTube videos). They used knowledge mined from webscale text copora to determine the best likelihood of various combinations of subject-verb-object triplets. They use a template-based approach to present the textual description, as: "Determiner (A, The) -Subject -Verb (Present, Present Continuous) -Preposition (optional) -Determiner (A, The) -Object." They evaluate the system automatically and by human evaluation.

Another interesting works were presented by [START_REF] Guadarrama | Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition[END_REF], [START_REF] Das | A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching[END_REF].

2-Sequence learning approach (A. [START_REF] Rohrbach | Movie description[END_REF], [START_REF] Donahue | Long-Term Recurrent Convolutional Networks for Visual Recognition and Description[END_REF], (J. [START_REF] Xu | MSR-VTT: A Large Video Description Dataset for Bridging Video and Language[END_REF], [START_REF] Venugopalan | Translating Videos to Natural Language Using Deep Recurrent Neural Networks[END_REF]: Known also as deep learning, probabilistic or data-driven approaches. This approach directly learns to map between video content and textual sentence. This approach can be mainly divided into two stages: a. In general, a sequence learning approach eludes the two steps of content identification and sentence generation by learning to match directly videos frames to human sentences. Different combination of encoding-decoding algorithm may be used, to mention CNN-RNN, RNN-RNN, and deep reinforcement networks. An example of a common architecture for video captioning using sequence learning approach is given in An example of video captioning architecture using sequence learning approach, taken from (Z. [START_REF] Wu | Deep Learning for Video Classification and Captioning[END_REF] Examples of some important works on sequence learning approach:

-Some of the researches on template-based video representation used statistical machine translations (Jeffrey [START_REF] Donahue | Long-term recurrent convolutional networks for visual recognition and description[END_REF], [START_REF] Barbu | Video In Sentences Out[END_REF], [START_REF] Kojima | Natural Language Description of Human Activities from Video Images Based on Concept Hierarchy of Actions[END_REF], (M. [START_REF] Rohrbach | Translating video content to natural language descriptions[END_REF]. These approaches map semantic sentence representation (e.g. key objects, locations, and scenes), with a Conditional Random Field (CRF) model, to high-level concepts such as the actors, actions and objects, for generating sentences. In [START_REF] Donahue | Long-Term Recurrent Convolutional Networks for Visual Recognition and Description[END_REF], the authors then improved their system by learning the output sequence representations into an LSTM model to translate it to a natural sentence. -In [START_REF] Venugopalan | Translating Videos to Natural Language Using Deep Recurrent Neural Networks[END_REF], the authors proposed an end-to-end neural network to generate video descriptions. By mean pooling, the features over all the frames are represented by one vector, to be used as an input of an LSTM model to generate sentences. For better modelling results, not only video contents and their spatiotemporal relationships were used, but also the syntactical structure. [START_REF] Venugopalan | Sequence to Sequence --Video to Text[END_REF] they extended their work by adding to the input frames and optical flow images to feed an encoder-decoder framework based on two LSTM modules. The encoding converts video into a compact representation, followed by the decoding to converts the output into a caption. -In [START_REF] Yao | Describing Videos by Exploiting Temporal Structure[END_REF], the authors proposed to utilize a temporal attention mechanism and a spatio-temporal convolutional neural network to obtain action features. The resulting video representations were used as input into the text-generating RNN. -In [START_REF] Pan | Jointly Modeling Embedding and Translation to Bridge Video and Language[END_REF], the authors proposed to model video content and textual semantics as a regularizer in Long Short-Term Memory architecture. [START_REF] Pan | Video Captioning With Transferred Semantic Attributes[END_REF] presented LSTM with transferred semantic attributes (LSTM-TSA) architecture where the semantic features were extracted from both images and videos using the CNN plus RNN framework for enhancing video sentence generation. -In [START_REF] Yu | Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks[END_REF], the authors used a hierarchical RNN (hRNN) to describe long video containing more than one event. The notion of hierarchical framework is to make use of the temporal dependency and semantic context between the sentences in a section. Mainly, they used two generators; a single sentence generator produced by a Gated Recurrent Unit (GRU) layer, using spatial and temporal information present in a precise time interval of a video, and a section generator models dependency between the sentences. As output, they generate a mundane description using multiple sentences in a section. -In [START_REF] Long | Video Captioning with Multi-Faceted Attention[END_REF], the authors proposed an LSTM with two multifaceted attention layers which export temporal, motion and semantic properties, using nearest neighbor (NN) search, Support Vector Machine (SVM) and hierarchical recurrent neural encoders (HRNE) for a subject and verb prediction based on the temporal features. -In [START_REF] Das | A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching[END_REF], the authors proposed to generate dense captions using sparse object stitching; their work for the description is not data-driven, however it is based on top-down ontology.

A comparative review of existing sequence learning approach in video description methods can be found in: (J. Xu et al., 2016), [START_REF] Ryoo | An Overview of Contest on Semantic Description of Human Activities (SDHA) 2010[END_REF], [START_REF] Awad | TRECVID 2018: Benchmarking Video Activity Detection, Video Captioning and Matching, Video Storytelling Linking and Video Search[END_REF], and [START_REF] Graham | Evaluation of automatic video captioning using direct assessment[END_REF]. An example of video description (dense captioning) is shown in Figure I-2. [START_REF] Zhou | End-to-End Dense Video Captioning with Masked Transformer[END_REF].

The colour bars represent different events. Coloured texts highlight relevant content to the event.

A comprehensive and interesting literature review on video description can be found in (Z. [START_REF] Wu | Deep Learning for Video Classification and Captioning[END_REF], and [START_REF] Aafaq | Video Description: A Survey of Methods, Datasets and Evaluation Metrics[END_REF].

Nevertheless, both approaches; Behaviour understanding and sentences generation and Sentences learning, have some major flows: 1-The Behaviour understanding and sentences generation approach: according to [START_REF] Venugopalan | Sequence to Sequence --Video to Text[END_REF] this approach is insufficient to model the richness of language used in human descriptions -e.g., which attributes to use and how to chain them effectively to generate a good description. Also, according to (Z. [START_REF] Wu | Deep Learning for Video Classification and Captioning[END_REF], the missing, erroneous and misidentified information extracted from the video frames leads to disjointed descriptions. In plus, the handcrafted templates risk being non-generic for the variety of scene types (J. Xu et al., 2016). 2-The Sentences learning approach: according to [START_REF] Aafaq | Video Description: A Survey of Methods, Datasets and Evaluation Metrics[END_REF] "The majority of current literature on video description focuses on domain specific short video clips with limited vocabularies of objects and activities". And so, current state of-the-art methods may not be suitable for long video sequences because they mainly focus on short topiccoherent ones. For that, the description of longer videos and scenes having variety of types remains a challenge, due to the need of large vocabularies and training data. In this domain, there is a lack of rich models that can learn the sentences to the appropriate features in the frames sequence.

Despite the tremendous work done in the field of video description in general, the existing state of art on video surveillance scene description as has it is particularity, still, however not deeply prospected. For example, video description for movies, broadcast news, or sports, can unveil practical drawbacks for video surveillance (Jiangung Lou, Qifeng [START_REF] Lou | Semantic interpretation of object activities in a surveillance system[END_REF], (C. [START_REF] Fernández | Determining the best suited semantic events for cognitive surveillance[END_REF]. But even though that most of the researches focus mainly on short non-surveillance videos, some of the advancements made in the approaches can be used in the field of video surveillance.

Next we mention some of the most influencing works on video surveillance understanding and description: -In Remagnino et al. (Remagnino, Tan, & Baker, 1998a), [START_REF] Remagnino | Multi-agent visual surveillance of dynamic scenes[END_REF], the authors mainly focus on traffic scene to represent the behaviour of pedestrians and vehicles, where their system is based on Bayesian network to give annotations for some events in natural language. They handle also some cases of interaction between two objects when the distance between the two is below a threshold.

-In [START_REF] Aishy | Object and Event Extraction for Video Processing and Representation in On-Line Video Applications[END_REF], the authors proposed a system for object and event extraction for video processing and representation. They mainly targets videos having realistic environment (with objects occlusions, artefacts). They proposed three processing levels: video enhancement, video analysis and meaningful content extraction (spatio-temporal features), and video events interpretation. They tested the system on real-time videos. Nerveless they did not work on textual description, but they highlighted in their approach, many interesting aspects in this field, especially concerning the real-case features extracted, and the logical relations.

-(Atsuhiro Kojima et al., 2002), (A. [START_REF] Kojima | Generating natural language description of human behavior from video images[END_REF] are two of the early works that proposed, on human activities, generating a hierarchical concept of actions for natural language description appearing in real image sequences. The authors primarily describe videos of a one person performing a single action. They detect humans head and hands by a probabilistic approach, and then they use their positions and the head direction to estimate the human posture. Meanwhile, the most appropriate verbs and many syntactic elements are selected. As last step, they used machine translation method to generate natural language text.

-In (Jiangung [START_REF] Lou | Semantic interpretation of object activities in a surveillance system[END_REF], the authors propose an approach for semantic interpretation of pedestrian and vehicle's behaviours for visual traffic surveillance. The trajectories recorded are then analysed using dynamic clustering, they introduce, based on HMM, a trajectory segment analysis method to every trajectory class. Then, in each segment, they assign the action of the tracked target to four basic types: Move Forward, Turn Right, Turn Left and Stop. Then they perform classification to feed the natural language semantic interpretation. For that, they use a simple grammar rule template: (The Obj) (Action) in (The place name) [at (high/low/middle) speed]. The system output module is only activated when: 1. A new action is occurring (Move Forward, Turn Right, Turn Left and Stop).

2. The object is entering a new region 3. An abnormal event is occurring Their system is restricted to one scene type, and one object type, and did not take interactions into consideration.

-In (C. [START_REF] Fernández | Determining the best suited semantic events for cognitive surveillance[END_REF][START_REF] Fernández | Interpretation of complex situations in a semantic-based surveillance framework[END_REF], the authors present a supervised ontology-based methodology. Their ontology shown in [START_REF] Fernández | Interpretation of complex situations in a semantic-based surveillance framework[END_REF] present interesting ideas and intersect with our ontology in many concepts. They first perform image segmentation for agent trajectories detection, body postures, and facial expressions, and targets identification. Then these information passes by a user for data filtering. They made this data for each detected agent available within a ground-plane representation of the controlled scenario. The uses XML for data exchange among the modules. Their approach considers, for video surveillance, different scene type, indoor and outdoor scenarios. Their proposed taxonomical events include basic actions and events (e.g. walk, run, turn) and some scenario-specific interpretation of behaviours (e.g. meeting, giving way, chasing). Their textual output is presented like: turn (Agent 20, left, crosswalk).

-In (Z. [START_REF] Xu | Semantic annotation of traffic video resources[END_REF][START_REF] Hu | Video structural description technology for the new generation video surveillance systems[END_REF], (Z. [START_REF] Xu | Video structured description technology based intelligence analysis of surveillance videos for public security applications[END_REF] the authors proposed approaches were called as Video structural description VSD. VSD targets at describing video content in text sentences. Firstly, they extract the semantic content from the video relying on spatiotemporal segmentation, feature selection, object recognition. Secondly, VSD aims at organizing resources in the video according to their semantic relations. The proposed method is based on ontology; which, between barracks, is highly recommended for a video structured description, it defines a number of concepts including vehicle, people, and traffic sigh, and their spatial-temporal relations, which allow users to annotate traffic events. In their approaches, they did not consider objects interactions description in the scene.

-A very recent and interesting work was presented in [START_REF] Ahmed | Natural Language Description of Surveillance Events[END_REF], where the authors present template-based technique that generates natural language descriptions of surveillance events. First, they track moving objects, and then they perform classification using CNN on the output into four classes: pedestrian, car, Bike, and Cycle. Finally, their system generates natural language description based on template: "A {color} {size} {type} in {speed}, coming from {entry zone} toward {exit zone}", as "A white medium vehicle in normal speed, coming from Main Building toward Residential Zone". Two important points were noticed, concerning the authors and their "experts" insist on the importance of structured templates, and human experts' assessment. Their system assumes a surveillance scene with some prior region information, and they did not consider interaction.

Other interesting research is presented in [START_REF] Tu | Joint video and text parsing for understanding events and answering queries[END_REF], (L. [START_REF] Xu | A Video Structural Event Description Model for Traffic Surveillance System[END_REF], (W. [START_REF] Hu | Semantic-Based Surveillance Video Retrieval[END_REF], [START_REF] Gerber | Deriving Textual Descriptions of Road Traffic Queues from Video Sequences[END_REF].

I.2. Goals and Challenges

Our primary goal, in this thesis, is to describe textually video surveillance scenes, in a comprehensive way to support police incidents reports. We focus on scenes containing exactly two objects. The secondary goal will be extracting valuable features useful for generating alerts and investigating intelligently the archives by surveillance system operators.

Automatic systems that can assist police and law enforcement agencies still need improvements in order to cope the existing needs when working with video surveillance. While the video analytics companies focus on big in appearance deliveries dissipating small basic issues that are the real police needs, the research field suffers from the missintegration, discontinuity of researches and missing the accuracy needed from the field (real cases); and the managers of such systems struggle from lack of knowledge in the "how", "how much" and "what" they really need to assist their systems in an efficient way.

The path leading to achieve these goals is vast, and contains many details. These details are with significant challenges and involve questions that need to be answered. Therefore, to direct this research, we could sum up the goals by asking the following:

The first question is about the best approach for a good video surveillance semantic representation.

Which one of the two approaches is best suited for video surveillance? Behaviour understanding and sentences generation approach or sentences learning approach? It depends on several points. -The sentences learning approach is still a developing field. It works well with images, but has limited vocabularies of objects and activities in videos, and is still not completely suitable for dealing with the variety of scene types and the long video sequences [START_REF] Aafaq | Video Description: A Survey of Methods, Datasets and Evaluation Metrics[END_REF][START_REF] Chen | Video to Text Summary: Joint Video Summarization and Captioning with Recurrent Neural Networks[END_REF]. In plus, in our research, there is an indispensable need for structural description and behaviour understanding features, like speed, trajectory, direction, shape and others; and working with learning approaches, till now, did not encounter all these aspects together. However, we believe that this evolving field will meet finally all the needs.

-For the Behaviour understanding and sentences generation approaches, as mentioned before, there is three main difficulties:

1) The handcrafted templates risk being non-generic for the variety of scene types. However, on video surveillance researches, one big advantage is that we knows exactly the needed output templates for the system; as, it is similar to the real case reports that already take into consideration the scenes varieties.

2) It is insufficient to model the richness of language used in human descriptions. Again, in the surveillance field, it is sufficient for the video surveillance reports to have a simple structured sentence as output.

3) The missing, erroneous and misidentified information extracted from the video frames leads to disjointed descriptions, which means the semantic content identification approaches for extracting all the needed features still not up to norm, and need a lot of improvements. Dealing with that, many enhancements appeared recently on many levels in computer vision field, especially after benefiting from the rapid and increasing machine learning field. And so, focusing on improving the semantic content extraction with machine learning, and then combining their advantages with the advantages of handcrafted features (X. [START_REF] Wu | Interpretable Video Captioning via Trajectory Structured Localization[END_REF], [START_REF] Cilla | Human action recognition with sparse classification and multiple-view learning[END_REF] it can improve the resulting content extraction.

Dealing with video surveillance system, from our perspective, requires many improvements to be made on many levels. But, in no case, we should lose the content understanding outputs because it is the main core of video surveillance analysis.

Therefore, a description system suitable for video surveillance can be built using the behaviour understanding and sentences generation approach. However, building a good system cannot be without making some improvement, on different levels, by taking advantages of the emerging machine learning field.

Many other questions are important and essential for this research. However, solving and answering them, is significantly challenging. a) How should the video description system be built in order to be more efficient and useful for different surveillance systems? b) How to decide what visual information to extract from video? c) What should a system describe? How to decide when generating a textual description along the time dimension? How much the description is practical and responds to the user needs? d) How to design a powerful sentence generation model? What an adequate textual/sentence representation contains? What is the best combination of different components of a representative sentence?

The following chapters will be answering these questions and a summary of these answers is presented in the chapter VI section VI.2.

I.3. Thesis outline

This section provides an outline of the entire thesis, which mainly consists of the next five chapters in order to achieve our goals.

In chapter II, trying to enable more integration between the high level semantics into the low level features automatically extracted, we present a new generic knowledge-based ontology, the "Video-Surveillance-Description Ontology", for describing video surveillance scenes.

As one of the most important features that can rule the way that an object can do the action, interaction or reaction, is its deformability, we present a method to classify, in chapter III, the deformable/non-deformable nature of a video object, using heuristic approach.

In chapter IV, we present our approach for textual description of surveillance scenes containing mainly two objects with main focus on the interaction occurring between the two objects. For this, we present how we produce activity matrixes of useful characteristics which can be used for generating alerts and querying the scenes, and how to generate textual descriptions of these matrixes.

In Chapter V, we highlight, based on our research and practical experience, the existing gaps between the surveillance systems operators' needs from one side, and the research field and the commercial (industry) field from the other side. Consequently, we present many propositions about how to address these drawbacks.

Finally, in Chapter VI, a general conclusion and future works of this thesis are presented.

II. Generic video surveillance description Ontology

II.1. Introduction

Multimedia content, particularly videos, are big data source. While current video browsing methodologies are mainly time-based, there is a crucial need to develop intelligent methods for effective storing, indexing, organizing, mining and retrieval from surveillance video databases. However, until now, there's still a lack in such automatic intelligent systems.

One probable reason for this lacking is that video is subject to different interpretation and description which can vary according to systems operators needs and applications [START_REF] Pavlidis | Why progress in machine vision is so slow[END_REF], [START_REF] Kunt | Comments on "Dialogue," a series of articles generated by the paper entitled "Ignorance, Myopia, and Naiveté in Computer Vision[END_REF], [START_REF] Jain | DIALOGUE Ignorance, Myopia, and naivete in computer vision systems[END_REF]. Many video representation techniques addressed this problem by trying to develop a specific solution for each application. Others focus on solving complex situations by assuming a simple environment, for example, without object occlusion, noise, or artefacts.

Consequently, advanced content-based video analysis has become a vastly active research field, and significant results have been reported for the last two decades [START_REF] Hua | Automatic music video generation based on temporal pattern analysis[END_REF], [START_REF] Muller-Schneiders | Performance evaluation of a real time video surveillance system[END_REF], [START_REF] Wactlar | Informedia-ii: Auto-summarization and visualization over multiple video documents and libraries[END_REF]. However, the lack of precise and generic models for video content representation and the complexity of video processing algorithms make the development of fully automatic video semantic content description a challenging task. Actually, the complexity and diversity of video scenes makes hard to map the low-level features extracted automatically from video data, into high-level semantic concept. This challenge, which often referred as the semantic gap, is corresponding low-level spatio-temporal features that can be automatically extracted from video data with high-level semantic concepts. This, causes the existing systems and approach to be too non-flexible and cannot satisfy the need of video applications at the semantic level. So the use of domain knowledge is very necessary to enable higher level semantics in automatic parsing. This is where "Ontology" enters the scene.

Ontology is composed of a set of terms (vocabulary) and specifications about their meanings (properties, relationships). The most referenced definition of the notion of ontology is given by [START_REF] Borst | Construction of engineering ontologies for knowledge sharing and reuse[END_REF] as: "a formal specification of a shared conceptualization". It was used, in many fields, as a knowledge management and representation approach. For the expression of concepts and relations in ontology, several standard description languages have been defined, we mention: Resource Description Framework (RDF) (RDF, 2004), Web Ontology Language (OWL) (OWL, 2004) and, for multimedia, the XML Schema in MPEG.

Ontology is a way to represent formally the knowledge. On the top of that, it is not qualified by the vocabulary but the conceptualizations that the vocabulary terms are intended to deliver. Thus, no change is conceptually made when translating the terms from one language to. In addition, ontology is a mean for the experts of different domains to communicate together, to share their experience and accumulate knowledge.

Many important efforts, based on ontologies, have been done in the field of video analysis, in general, and video surveillance in particular. In the state of the art, we present some of these works.

In the last section of this chapter we present our ontology named "Video-Surveillance-Description Ontology". It is a new generic approach for video surveillance description, and designed to be used as a generalist high-level layer for video analysis, principally in a video-surveillance system. We considered the temporal dimension of the video, using appropriate features. Our proposed ontology introduces six main classes; one of which is a representation for generic scene types, divided into fifteen subtypes according to the number of moving objects before and after the interaction. This ontology will be based on in the next chapters to fulfil an automatic textual description of video surveillance, focusing mainly on interactions between objects.

II.2. Related Works

II.2.1. Ontology benefits and requirements

Ontology is a way to reduce the semantic gap in video analysis between low level features and the needed output. They present powerful mechanisms for organizing, structuring, and sharing knowledge. The reasoning, flexibility, share-ability, and representation make these models suitable to surveillance domains. For instance, ontology of video understanding will enable different experts to communicate and exchange their point of view about functionality or an expected output result.

We can already see that having a domain-based ontology is important to reach our objectives. As Korpipää, et al. mentioned, some of the basic requirements that hold also for our approach when designing our ontology are, the simplicity, the flexibility, the extensibility, the genericity, and the expressiveness [START_REF] Korpipää | Managing context information in mobile devices[END_REF].

Despite the great advancements in the last decade, the complexity and the quantity of possible complex activities [START_REF] Naeem | A comparison of two hidden markov approaches to task identification in the home environment[END_REF], the importance of the semantics associated with a behaviour (L. [START_REF] Chen | Ontology-based activity recognition in intelligent pervasive environments[END_REF] and the interaction of several objects in the same environment [START_REF] Cook | Ambient intelligence: Technologies, applications, and opportunities[END_REF], [START_REF] Singla | Recognizing independent and joint activities among multiple residents in smart environments[END_REF], among others, make creating a suitable ontology, based on understanding of human behaviour, a challenging task.

II.2.2. Previous works on video analysis using ontologies

The state of the art of both approaches for video analysis, the data-driven approaches and the knowledge-based approaches, mentioned in the state of the art section I.1, reports many researches based on ontologies, [START_REF] Rodríguez | A Survey on Ontologies for Human Behavior Recognition[END_REF].

Nevertheless, working on ontologies trespasses the video surveillance domain to a wider one which the video in general.

II.2.2.A. Ontology on contextual information and context-aware

A significant amount of researches on ontology has been done for the structural representation and recognition of contextual information [START_REF] Rodríguez | A Survey on Ontologies for Human Behavior Recognition[END_REF], and activities and interactions. Important context-aware ontologies have been proposed, like, CONON (CONtext ONtology) [START_REF] Haas | Reprints available directly from the publisher Photocopying permitted by license only[END_REF], the Pervasive Information Visualization Ontology (PIVOn) [START_REF] Herrera | A fusion approach for managing multi-granularity linguistic term sets in decision making[END_REF], the Context Aggregation and REasoning (CARE) middleware [START_REF] Herrera | A model based on linguistic 2-tuples for dealing with multigranular hierarchical linguistic contexts in multi-expert decision-making[END_REF], and the fuzzy ontology (B. [START_REF] Wang | A Normalized Numerical Scaling Method for the Unbalanced Multi-Granular Linguistic Sets[END_REF]. A wide range of factors are used to classify these previous work in human motion analysis and video understanding, such as: model-based vs. non-model based, human-object interactions and group activities, action and activity recognition and classification, complex activities recognition and behaviour understanding, and video description, etc. Also, a number of surveys have reviewed the use of ontologies for context modelling, user context and human Behaviour [START_REF] Rodríguez | A Survey on Ontologies for Human Behavior Recognition[END_REF], [START_REF] Villalonga | Ontology-based high-level context inference for human behavior identification[END_REF].

II.2.2.B. Ontology in the domain of video surveillance

For video structured description, ontologies are highly recommended (Z. [START_REF] Xu | Semantic annotation of traffic video resources[END_REF]. In the domain of video surveillance, various approaches of ontologies and algorithms were used to address different stages of the problem. [START_REF] Vezzani | Video Surveillance Online Repository (ViSOR): An Integrated Framework[END_REF].

Video surveillance has its own set of most significant entities, terms, hierarchies, and relations. Due to the huge set of possible cases combined with the flexibility of description, the definition of unique video surveillance ontology is very ambitious and probably unfeasible. Nonetheless, a set of actions, events and entities can be selected due to their importance. The surveillance community has made some proposals for action, event, human activity and behaviour ontologies. Some shared concepts can be found among the following ontologies; also some ideas intersect with our proposed ones:

Video Surveillance Online Repository (VISOR) [START_REF] Vezzani | Video Surveillance Online Repository (ViSOR): An Integrated Framework[END_REF]) is a platform for annotating, and retrieving surveillance videos, which used as a support tool for different projects. It contains a large set of multimedia data and corresponding annotations. VISOR provides a list of video surveillance concepts used in the Visor system. The main concept of dividing between context and content is shared between many ontologies, including ours.

In [START_REF] Ly | Specific Behavior Recognition Based on Behavior Ontology[END_REF] a behaviour ontology is proposed, mainly based on set of scene model related by set of time relation. The set of scene model contains set of object model where low level data is specified, set of object relation, and set of object condition. Some of the concepts of the object Model intersect with ours.

More recent work can be found in (Alonso, Leal, Escalante, & Succar, n.d.), The authors present ViVA ontology, which is based on [START_REF] Kazi Tani | Events Detection Using a Video-Surveillance Ontology and a Rule-Based Approach[END_REF], [START_REF] Sanmiguel | An Ontology for Event Detection and its Application in Surveillance Video[END_REF]) and ("VISOR," 2017). ViVA ontology proposes three main classes Content, Context and System. VIVA was designed with owl format and using Protégé. Protégé is a suitable tool for ontology presentation, which we decided to use it for our approach, in the interpretation of my graphical representation. Also, concepts concerning place, weather, location, and object may meet our same objectives, as follow; some of those influence our ontology.

Other interesting ontologies can be found in the Appendix VIII.1.

II.3. Proposed Ontology

The varied nature of objects participating to a scene, the variety of scene types or contexts, and the complex nature of the object behaviours, actions and interactions and in the execution, requires an abstract level of information to reduce the size of the description scope. This work presents an ontology-based method that combines low-level primitives of objects basic features, like size, colour, locations, speed and others, that should allow to intelligently deriving more meaningful high-level information.

In order to realize the knowledge-based and automatic generic description of video surveillance introduced in the previous section, the knowledge for video analysis is abstracted. Among many distinctive characteristics for this ontology, we mention that it:

1-Focuses mainly on the objects interactions, nonetheless it is expendable.

2-

Presents detailed propositions about the interaction, from a methodologic and systematic approach.

3-

Is not directed by the results of the automatic analysis, and there is no preassumption or condition which restricts this ontology.

4-

Targets mainly the level of generic and abstract description, but it can be applied to any scene type or context.

5-

Shall be convenient to describe real interactions during incidents as they appear in CCTV control rooms' reports.

6-

Focuses on new concepts concerning mediation, action at a distant and close interaction, deformable and non-deformable objects, and others.

Our proposed ontology, named "Video-Surveillance-Scene-Description Ontology" or "VSSD Ontology" mainly describes the concepts that relate video, objects, and actions. VSSD ontology has been designed to be used as a generalist high-level layer for a video analysis, principally in video-surveillance system. VSSD ontology proposes six main classes: Context, Object, Video, Activity, Scene and Descriptor (see Figure II-1).

II.3.1. Context

This class contains all the elements that provide information about the real context, see Figure II-2. For example: the GPS coordinates, the place where the action happens which can have two types: (Indoor: Bank, School, etc.; Outdoor: circulation, garden, Parking, etc.); the environment (weather, altitude, temperature, pressure, lighting, humidity, noise) and the time class, one of the most important classes that drive all other class.

II.3.2. Object

The Object class represents instances of humans, animals, plants, machines and all other inert objects, see Figure II-3. This class can represents all what exist in an environment. One of the most important features that can rule the way that an object can do the action, interaction or reaction is its deformability. The deformability criteria is mainly deduced from the object shape and motion, and is based on the degree of deformation (Kambhamettu C., Goldgof D. B., Terzopoulos D., Huang T. S., 1994). When focusing on an area of interest, the first thing to distinguish, if an object appears far/deep in the frame, is its deformability. Non-deformable objects actions or reactions during an interaction are easy to detect, analyse, understand and maybe predict. When deformable parts of an object, move freely in unpredicted way, the prediction becomes more difficult even for a human brain.

We chose to group all objects in two general sub-classes, deformable and nondeformable objects. Object deformability dilemma is considered in chapter III, and in section IV.3.2 object classification. For example, humans and animals are ''deformable''. Plants, machines and inert objects can be deformable or non-deformable, e.g. a tree is considered as deformable when each of the branches can move differently than the others or the trunk. A machine in this ontology indicates the machines controlled by intelligence (humans or artificial intelligence) like a car or a robot, in the opposite of an inert object like a box. We may find some of those objects, in other ontologies, under the name of agent.

II.3.3. Video

In visual surveillance systems, the cameras are mainly fixed. As the same object may appear several times in the same video, each appearance will be considered as an instance in Video_Object class. So, the Video_Object class is a subclass of video and object classes, see Figure II-4. This instance is delimited from the first moment of that appearance to the last one. A sub_object is mainly used for deformable objects, for example for articulated segments of human and animal bodies, or parts of machines, etc. A video object can have several sub_Objects. As we may have many states for each appearance (instance of video_object class), each of the states describes the object/video_object state. Similarly many of the states can be taken for each of the sub_objects to create a sub_object_state, see Figure The number of states depends on the time of appearance, time of disappearance, and the suitable frame difference that we would take. In plus, for each state, the video object state can have many features (attributes) like shape, surface, displacement, speed, trajectory and many others.

II.3.4. Activity and Action

Different taxonomies are used for describing an action. We can find, among others, the terms operation, gesture, action, event, activity, and behaviour. So far, there is not a unique standard ontological definition of those notions or concepts. Many can be found in different articles [START_REF] Herath | Going deeper into action recognition: A survey[END_REF], [START_REF] Ranasinghe | A review on applications of activity recognition systems with regard to performance and evaluation[END_REF], [START_REF] Morris | A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance[END_REF], [START_REF] Lavee | Understanding video events: a survey of methods for automatic interpretation of semantic occurrences in video[END_REF], [START_REF] Kaptelinin | Activity Theory[END_REF].

Usually, the literature names what human is doing and the way it is doing it human behaviour or human activity interchangeably [START_REF] Ros | Online recognition of human activities and adaptation to habit changes by means of learning automata and fuzzy temporal windows[END_REF]) [START_REF] Remagnino | Ambient intelligence: a novel paradigm[END_REF], [START_REF] Rashidi | Keeping the resident in the loop: Adapting the smart home to the user[END_REF]. These activity/behaviour terms correspond to a sequence of human actions. However, most of these authors agree to define human action as the simplest unit in the human activity. As new approaches are being developed (L. [START_REF] Chen | Ontology-based activity recognition in intelligent pervasive environments[END_REF], new levels appear in the system. For that, a difference should be made between the terms human behaviour, events and activity to differentiate between the concepts of what a human is doing in the environment (activity), and the purpose or meaning it could have (behaviour). An Event is the occurrence of an activity in a particular place during a particular time interval. The Behaviour is a description of activities and events within a specific context.

In our ontology we embed three hierarchical layers: activity, action-interaction and operation.

An activity, according to [START_REF] Blunden | Leontyev's Activity Theory and Social Theory[END_REF], is the units of life. It is purposeful and developing interaction between actors ("subjects") and the world ("objects") [START_REF] Kaptelinin | Activity Theory[END_REF]). An activity is hierarchically structured into actions, see Figure II-6. For more complex scenes, activities may be, sequential, or concurrent according to performing time.

The second layer is the Action. The action is based on conscious processes concentrating to fulfil a goal or its sub-goals. In the philosophy of action [START_REF] Wilson | Action[END_REF], an action is defined as intentional, purposive, conscious and subjectively meaningful activity. For example, pushing a person is an action, while catching a cold is not considered a one.

In case of two or many objects, an action begins when one of those objects has the intent to perform an action, even while approaching. This action ends when the objects retreat. They may approach again to begin another action.

Another important concept is the mediation. The main distinctive features of humans, such as language, culture and society, the production and use of advanced tools, etc., all involve mediation; here we note the mediation of information as the most important one among interactions. They represent different aspects of the same phenomenon, that is, the emergence of a complex system of structures and objects, both immaterial and material which serve as mediating means embedded in the interaction between human beings and the world and shaping the interaction. In cultural-historical psychology, mediation is, arguably, the most important concept of all; it serves as the cornerstone of the activity theory as a whole [START_REF] Vygotsky | Mind in Society[END_REF]. An example for the mediation is a human shooting another object (human, animal…). In this case the bullet can be considered as the mediation. We may equally well consider the linguistic interaction as a transmission of information, for example saying "Hello". In the opposite, when two humans are boxing, two animals are fighting, or when two animals are following each other, there is no mediation between the two objects or unmediated action.

We can consider that implicit information helps both objects to coordinate their interactions.

In the case of one, two or many objects, and where the action/interaction is unmediated, or at least not well noticed as visual mediation by the application, we distinguish between two action types: a-There is no physical contact: then we consider "action at a distance" or "far action/interaction", for example: when two objects are running together, or when two humans are saluting each other, etc. b-There is physical contact: then we consider "close action/interaction", for example, when two humans are fighting, or hand shaking, etc.

An Action is a series of operations done by an object on nobody, object, or many objects. The operations are considered the lower-level units implementation of the action. Accordingly the Interaction States can document the state of interaction at a related moment (existence, type and aggressiveness). We present the relations between components and action. But those relations can be the same for activity and operation, or for the interaction state. We mention that: -An object or video_object or sub_object can have an action/interaction, and an action is done by an object or video_object or sub_object. -A video contains an action, and an action is viewed in a video.

-A video_object_state or a sub_object_state is a part of an action, and an action can have instance a video_object_state or a sub_object_state.

II.3.5. Scene

To define a methodologic and systematic approach to describe the video scene especially the interaction between video objects in video surveillance, we identify fifteen types according to the number of moving objects and to their characteristics (features) before and after the action.

1. 0 Object (Scene without any moving objects): when no object is moving in the scene. 2. 1 Object  0a (Single object stops, no interaction with the environment): when a single object is moving in the scene at some moment it stops. Examples: car parks, etc.

3. 1 Object  0b (Single object stops, interaction with the environment): when a single object is moving in the scene, without any interaction with another moving object, at some moment it stops, after mainly changing and interacting with the environment (background). Examples: car hits a store causing it to stop, etc. 4. 1 Object  1a (Single object, no interaction with the environment): when a single object is moving in the scene without any interaction with another object or without changing anything in the environment (background). Examples: human walking, or doing sports, car passing, etc. 5. 1 Object  1b (Single object, interaction with the environment): when a single object is moving in the scene without any interaction with another moving object but mainly changing and interacting with the environment (background). Examples: person switching on the lights, person is smoking, car switching on the lamps, crashing an ATM machine etc. 6. 1 Object  1c (Single object, interaction with the inert objects of the environment):

when a single object is moving in the scene without any interaction with another moving object but changing and interacting with the inert objects of the environment (taking or leaving an inert object); and by that it changes its characteristics either gaining (good influence) or losing (bad influence) some. Examples: person or animal handling an inert object like box, person wears or removes his vast or hat, etc. 7. 1 Object  1d (moving object trigger an inert object and stops): when a single moving object in the scene, at a given moment, performs an action with another inert object, hence the object stops and makes the inert object to move. Examples: one ball hit another fixed ball and stops, one moving car hits another car hence it stops and makes the other car to move, etc. 8. 1 Object  2a (moving object trigger an inert object): when a single moving object in the scene, at a given moment, performs an action with another inert object and makes it to move. Examples: one ball hit another fixed ball, person is opening a door, one moving car hits an inert object (like another car) and makes it to move, etc. 9. 1 Object  2b (moving object divides into 2): when a single moving object in the scene, at a given moment, divides into 2 objects. Examples: person jumps out from a car, person removes his vast or hat, etc. 10. 2 Objects  1a (moving object stops another moving object): when there are two moving objects in the scene and, at a given moment, one object do an action and stops the other object. Examples: a moving car hits a moving person, etc. 11. 2 Objects  1b (2 moving objects merge into 1): when there are two moving objects in the scene that, at a given moment, merge into one single object. Examples: person jumps into a moving car, a person jumps on a moving skateboard, a person picks up and wears a hat, etc. 12. 2 Objects  0 ( 2 moving object stops after interaction): when there are two moving objects in the scene that, at a given moment, interact and stop moving. Examples: two cars make an accident; two objects collide and stop, etc. 13. 2 objects  2a (2 moving objects without interaction): when there are two moving objects in the scene without any interaction between them. Examples: two cars passing near each other, two humans passing by without any far or close interaction, human and animal co-appear in a scene without any kind of interaction, etc. 14. 2 Objects  2b (2 moving objects with interaction): when there are two moving objects in the scene, at a given moment, they interact, and then continue. Examples: two cars are passing near each other trying to avoid collision, two humans follow each other, two humans walking together, animal walking near a human, two humans salute each other, two humans waving to each other, two humans seeing and walking toward each other, animal is enclosing on a human, two animals fighting, two humans boxing, etc. 15. Many Objects  Many Objects (Group of moving objects with interaction): when there are many moving objects in the scene, interacting together at a given moment, and continuing after. We do not consider here many objects in the scene so that the interaction can be divided in couples. This category it meant to describe scenes with a crowd. Anyway, this category may be divided into many other ones, but as it is not our field of interest, we preferred to keep it as one category. Examples: group fighting, or cheering, etc.

These fifteen types are mainly focused on scenes with 0, 1, and 2 objects in the scenes. For more than two objects in the scene, we put all of them in one class for later reconsideration. We must notice that a scene can also be a mixture of many of these types.

Concerning the Scene_Sub_Type, we may introduce more detailed interaction categories, such as: At distance or physical, Aggressive or Peaceful.

II.3.6. Description

This class is intended to describe the whole scene from objects to action/interaction and context, according to the scene type and sub_type. It contains two main sub_classes: Abstract_Description, Semantic_Description, see Figure II-7. Those descriptions of a scene can be done using two methods: 1-Holistic method: this method takes the whole scene as one single closed box. It does not require for example the localization of body parts, the object or the action identification; the most important is what happens. Using this method, we consider all the possible combinations of actions/interactions in order to recognize, later, which one is the closest to this scene action. It is considered that the actuator actuated and action as a single box. 2-Detailed method: it is the study of each element of the scene, where the identification of each object, sub-object, action, operation, element apart is required.

Then, the scene description, according to the scene type and sub-type and the method used, can be a generic abstract (context free) or a much more semantic text where the context has a big influence. In Figure II-8 we present the abstract description used in this study, see chapter IV. To have a semantic description one can add, simply, on this abstract description the information taken from the context, like location, time, place, and place, etc. For example: "At frame 201: "Deformable" object "1" enters the scene, in "C" spot, on the "Left Middle" of the "Outside" area of the camera field of view, heading "Up Left", having respectively "regular" shape, "small" surface, and "slow" speed".

"On Saturday 10/11/2018, at 11:35:22, a person "1" enters the scene, in the intersection " 35.484180), on the right of Verdun street, heading south, having respectively small body, and slow speed".

Example of abstract description

Example for semantic description

II.4. Conclusion

In this chapter, we presented our proposed generic ontology for video description, mainly for video surveillance, taking into consideration some shared concepts as context, object, sub_object, activities, etc. Also, it presents some entities with new concepts like deformable/non-deformable object, fifteen scene types, close/far interaction, aggressiveness of interaction, etc. This ontology will be based on, in the next chapters, to fulfil an automatic textual description of video surveillance, focusing on interactions between two objects.

III. Classifying deformable and non-deformable video objects III.1. Introduction

For the purpose of semantic video analysis and understanding, it is especially important to recognize and study the video content-i.e., the background, objects, actions, and their movements-to better understand their meaning. Recent research focuses on object movement and its meaning. Accordingly, object properties and characteristics are of considerable importance. One property that significantly drives and influences the objects movements and actions is the object deformability. Object deformability is an important property to qualify the actioner and the actionee, it also gives the main clues to well understand the action. From surveillance point of view, non-deformable objects reactions, during an interaction with another object, are easy to detect, analyse, understand and maybe predict. While deformable parts of an object, make the analysis more difficult even for a human brain. As the deformability criterion of an object is one of the most important high-level features, we found it crucial to differentiate between the two classes.

In many research works, object deformability is a mandatory prior piece of information, for further interpretation, which is not actually automatically extracted, instead it is assumed.

A deformable object is an object that, when in motion, can undergo shape deformations, for example, a walking man, or a running animal. A non-deformable object, by contrast, has a rigid shape, for example, a passing car, an opening door. We define temporal motion as a fragment of an object motion for a small number of successive frames. "Non-rigid motion" is standardly used to refer to all articulated, elastic, and fluid motion, denoted here "deformable motion". Likewise, rigid motion is denoted as "nondeformable motion". Importantly, deformable objects can have both deformable and nondeformable motion, whereas non-deformable objects are restricted to non-deformable motion.

The deformable / non-deformable nature can hardly be established by a learning approach given the difficulty of producing the data necessary for learning. On the other hand, from a visual point of view, the definition of the concept is relatively well defined. So we propose a heuristic approach expressing a physical model. This chapter presents a new fully automated method for classifying deformable and non-deformable objects. It analyses the object's movements (object motion), differentiates deformable from non-deformable motion, and infers from this whether the moving object is deformable or non-deformable. Our classification method is effective without having any prior information about the environment, the shape of the object, or its displacement, and it does not depend on pre-assumptions. Our method aims mainly to deal with videosurveillance content where there is only one moving object in the scene. But applying object detection or segmentation algorithm, as done in the chapter IV, this method can easily be extended and applied on scenes having several objects.

As stated above, we study object deformability by analysing its motion. Thus, a motion-estimation technique is used to estimate motion between frames. Geometric transformations (viz., Fundamental matrix and Homography) are pursued to determine whether each of the observed motions corresponds to a transformation. Hence, it is indispensable that we investigate, in Section III.2, the background and related works in both motion and object deformability, in addition to some motion-estimation techniques and geometric transformations. In Section III.3, we explain our approach. We then present, in Section III.4, the experiments we have done in order to validate and evaluate our method.

III.2. Background and related works

III.2.1. Background

The world seen by humans when moving appears stable, rigid, and threedimensional (3D). This impression is probably the result of the fact that retinal images change over time [START_REF] Hogervorst | Monocular discrimination of rigidly and nonrigidly moving objects[END_REF]. A fundamental ability of the human visual system is its capacity to interpret motion in space. The visual system is capable of extracting useful information about the 3D structure from these retinal changes. This process is usually called structure-from-motion (SFM). However the ease with which humans detect motion and navigate around objects, and the difficulties in duplicating these capabilities in machines, have led to major challenges for computer engineers and scientists in understanding vision in humans and machines [START_REF] Aggarwal | On the computation of motion from sequences of images-A review[END_REF].

Human vision is privy to many sources of depth information that do not depend merely on stereovision. These sources of information include motion parallax, shape from shading, and textural information. Parallel to this, studies that work with photography in general (i.e., both video and images) have proposed methods for extracting useful information about objects from images and frames. We distinguish works in the following directions: -translation and/or rotation movement of a rigid body, [START_REF] Tsai | Uniqueness and Estimation of Three-Dimensional Motion Parameters of Rigid Objects with Curved Surfaces[END_REF].

-projection: affine or orthographic, or perspective, (Del Bue, Lladó, & Agapito, 2007a).

-dimensional approaches: 3D (Structure From Motion SFM, parametric…), or 2D (Optical flow, change detection…), [START_REF] Zang | Rapid inference of object rigidity and reflectance using optic flow[END_REF]. -extraction of 2D object features; points, corners, lines, edges, conic arcs, features correspondences, or the optical flow, [START_REF] Stoll | Adaptive Integration of Feature Matches into Variational Optical Flow Methods[END_REF]. -appearance of the object in multi-frame, [START_REF] Hogervorst | Monocular discrimination of rigidly and nonrigidly moving objects[END_REF].

-types of view: monocular or stereoscopic or multiple view images (R. Hartley & Zisserman, 2003a).

Good reviews and plenty of explications of the available methods for estimating the 3D structure and motion from sequences of monocular and stereoscopic images can be found in [START_REF] Aggarwal | On the computation of motion from sequences of images-A review[END_REF]. Similarly, [START_REF] Huang | Motion and Structure from Feature Correspondences : A Review[END_REF], provides an excellent review for exploiting the consistency by using the multi-frame analysis and studying the object motion and structure from feature correspondences. For a long time, studies proposing motion-based approaches to motion analysis have been largely restricted to the study of non-deformable object motion, or they were obliged to assume it. However, in the real world, deformable object motion is far more common.

Recently, the studies on analysing articulated motion, particularly human motion, has been inspired by a tremendous number of applications, and this analysis can be generally categorized as: (1) model-based approaches (J. [START_REF] Wang | Learning Actionlet Ensemble for 3D Human Action Recognition[END_REF] and (2) methods that do not require a priori shape models (H. [START_REF] Wang | Dense Trajectories and Motion Boundary Descriptors for Action Recognition[END_REF]. In the latter approach mostly useful for motion tracking when dealing with an unknown object where no a priori knowledge about the motion or the object's shape is available. The major difficulty to this type of approach is to establish feature correspondence. Consequently, to get around the problem, researchers either they impose constraints on the object's behaviour, or they focus on high-level processing supposing that matching is known a priori. Model-based approaches have the benefit of knowing, in priori, the approximate shape of the object, simplify this problem. However, these methods are not applicable when information about the object's shape is unavailable. Despite this, there has been a lot of work on non-deformable objects and object deformability, and an increasing number of studies on deformable objects, especially with regard to simulating or segmenting articulated objects. However, most of these works assume the deformability (or non-deformability) of the objects, and relatively little research concerns the automatic discrimination of deformable and non-deformable moving objects. Some of this research is mentioned in the following subsection (i.e., Subsection III.2.2). In addition, because our object classification technique is based on motion classification, we will briefly address motion-estimation techniques-viz., homography and fundamental matrices-given its pertinence to this study.

III.2.2. Works related to object deformability

L. [START_REF] Wixson | Classifying moving objects as rigid or non-rigid[END_REF]) used a reference image preobtained from the video sequence for classifying moving objects as rigid or non-rigid based on the similarity of their appearance over multiple frames. They took as hypothesis that the appearance of the rigid objects under viewing conditions similar to orthographic projection changes much more slowly than that of most of the non-rigid living ones. It should be mentioned here that feature correspondences are not used with this method. According to the authors, the results are preliminary, and the method requires further testing and quantification; and additional work is needed to mitigate fluctuations resulting from occlusions that occur when the object moves behind a structure and for dealing with small object movements. In addition, they use relatively few number of experiments compared to other studies. A. J. [START_REF] Lipton | Local application of optic flow to analyse rigid versus nonrigid motion[END_REF] used the residual flow to analyze the rigidity and periodicity of moving objects. His work is based on the assumption that rigid objects present little residual flow, whereas non-rigid moving objects display higher average residual flow. However, this method cannot be applied to slowly moving objects nor to any revolving objects. R. [START_REF] Cutler | Robust real-time periodic motion detection, analysis, and applications[END_REF] proposed a method based on the temporal self-similarity of a moving object. Their approach suggests that, when an object displays periodic motion, its self-similarity measure shows periodic motion. They use periodicity to categorize moving objects. But their technique assumes that each object can be properly segmented from the background. However, this assumption does not always hold true. J. Yan and M. Pollefeys [START_REF] Yan | A general framework for motion segmentation: Independent, articulated, rigid, non-rigid, degenerate and non-degenerate[END_REF]) concentrated on a factorization method based on motion segmentation in trajectory data. Factorization-based methods find an initial segmentation by thresholding the entries of a similarity matrix built from the factorization of the matrix of data points. According to E. Elhamifar and V. Vidal [START_REF] Elhamifar | Sparse subspace clustering[END_REF], it is likely that such factorization-based methods, in general, are correct provided that the subspaces are independent, but they fail when this assumption is violated. Moreover, these methods are sensitive to noise. Otherwise, a spectral-clustering method, such as the one used by J. Yan and M. Pollefeys, can be used to deal with the issues already mentioned by using local information around each point to establish similarity between pairs of points. The objective in J. Yan and M. Pollefeys was to segment a wide range of motion, including independent, rigid, non-rigid, articulated, degenerate, non-degenerate. The data is then segmented by applying spectral clustering to this similarity matrix. According to E. Elhamifar and V. Rene, such methods are less effective at dealing with points near the intersection of two subspaces, because the neighbourhood of a point can contain points from different subspaces. This issue can be resolved with multi-way similarities that capture the curvature of a collection of points within an affine subspace. However, the complexity of building a multi-way similarity grows exponentially with the number of subspaces and their dimensions.

A. Del Bue et al. [START_REF] Del Bue | Segmentation of rigid motion from non-rigid 2d trajectories[END_REF]) evaluated a method that uses a trajectory to automatically segment a set of rigid and non-rigid moving points within a deformable object, given a set of 2D image measurements. They noticed a higher misclassification ratio with weak perspective effects, and a greater proportion of non-rigid points. Furthermore, points that are rigid for only a part of the sequence may go undetected. In addition, their proposal was subject to a relatively few experiments. D. [START_REF] Zang | Rapid inference of object rigidity and reflectance using optic flow[END_REF] used the optical flow to infer the object's rigidity and reflectance. They used the optical flow exclusively to detect rigid object motion for both specular and diffuse reflective surfaces. However, in order to derive the relationship between optic flow and rigid-object motion, they assumed that both the viewer and the environment were distant from the object, approximated by orthographic viewing and illumination parameterized by the direction on a sphere. Further, their results are also based on relatively few simulation examples and experiments.

Feng et al. [START_REF] Feng | Rigid and non-rigid object image matching using deformable object image discrimination[END_REF] proposed an image matching method to match rigid object image and non-rigid object image by utilizing the same feature.

To the best of our knowledge, no previous research has proposed a method with the following characteristics: full automation in discerning deformable and non-deformable objects; complete generality and applicability to any type of object in a video (i.e., generalperspective projection for the general motion of a general object); a method that does not rely on conditions, assumptions, or additional information about the object in advance; one that takes into account the fact that deformable objects sometimes behave as nondeformable objects; and a method that benefits from temporal consistency. Our approach is the first one to join all those points together.

In this study, the discrimination of rigid from non-rigid motion is studied, to farther infer the rigidity or none of the object.

III.2.3. Motion estimation

Motion estimation, in a video sequence, is to determine the motion's vectors that describe the change from one frame to its adjacent one. As the motion is in 3D scenes, and as the images' frames are its projection onto a 2D plane, so finding the true motion is an illposed problem, so it called the apparent motion. The motion vectors may relate to the entire image or to specific parts, such as pixels, or even rectangular blocks, to build the motion field. In dense motion fields, each point is assigned a vector consistent of the motion direction, velocity, and the distance from an observer to the image location.

In video sequences, motion is a key source of information. Estimating the motion field is a useful starting point for solving several issues pertaining to motion analysis. Efficient and accurate motion estimation is essential to image-sequence analysis, motion analysis, computer vision, and video communication, and this information is fundamental to video understanding and object tracking.

The most common methods for estimating the motion field can be categorized into pixel-based methods (or "direct" methods) and feature-based methods (or "indirect" methods) [START_REF] Dufaux | Motion estimation techniques for digital TV: A review and a new contribution[END_REF]: Direct Methods: -Pixel-recursive algorithms -Transform-domain approaches -Optical flow [START_REF] Barron | Performance of optical flow techniques[END_REF], [START_REF] Beauchemin | The computation of optical flow[END_REF].

 Differential techniques.  Phase-correlation methods.

 Frequency-domain methods.

 Block-matching methods [START_REF] Khammar | Evaluation of different block matching algorithms to motion estimation[END_REF], [START_REF] Love | An Empirical Study of Block Matching Techniques for the Detection of Moving Objects[END_REF]). -Indirect Methods:

 Feature-correspondence methods (Farin & de With, 2005), [START_REF] Torr | Feature Based Methods for Structure and Motion Estimation[END_REF].

For more general comprehensive and comparative techniques the reader is referred to Appendix VIII.2.

III.2.4. Projective transformation and Epipolar Geometry

For better understanding how the transformations (the homograghy and the fundamental) can serve our objectives in this chapter, we found it indispensable to clarify some points:

III.2.4.A. Homography (projective transformation)

Homography is conceptually related to collineation, projectivity, and planarprojective transformation. It is an invertible transformation from a projective space (for example, the real projective plane).

It is considered to be a general transformation between the world and the image plane after imaging with a perspective camera (R. [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF]. Homography also describes the transformation from one plane to another (i.e., a mapping from P 2 → P 2 ). For example, the projection of points of a plane into an image plane can be described with homography.

Thus, for a set of point correspondences {X i ↔ X i '} in two images, if all the points X i are coplanar, then X i and X i ' are related by a non-singular 3×3 homography matrix, such that:

X i '= H.X i (eq. III-1 )
H can be represented with homogeneous coordinates, as a non-singular linear homogeneous transformation.

Various algorithms have been proposed to estimate homography. Some use point correspondences, while others use lines, lines and points, conics, curves, discrete contours, or the planar texture.

In general, estimation algorithms can be classified according to [START_REF] Criminisi | A plane measuring device[END_REF] as: non-linear geometric (non-homogeneous) estimations, linear nonhomogeneous estimations, and linear homogeneous estimations. For more information concerning these methods, the reader is invited to see [START_REF] Agarwal | A survey of planar homography estimation techniques[END_REF] and [START_REF] Dubrofsky | Homography Estimation[END_REF]: NB: In real images, the position of the points x i and x′ i is perturbed by noise. Image measurement errors will occur in both images and the estimation of H might not be perfectly accurate. For this reason, the image of x i in the first image is mapped by H to the point 𝐻. x i in the second image; and it is not necessarily equal to x′ i : x′ i ≠ 𝐻. x i ; vice versa : Further, for all the corresponding points x i ↔ x′ i / 𝑖 = 1 … 𝑁 :

x i ≠ 𝐻 -1 . x′ i .
𝐸𝑟 𝐻 = ∑ (𝑑(𝑥′ 𝑖, 𝐻. 𝑥 𝑖 ) 2 + 𝑑(𝑥 𝑖, 𝐻 -1 . 𝑥′ 𝑖 ) 2 ) 𝑁 𝑖=1
(eq. III-3)

Other error measurements can be used. These errors have been identified in the literature.

III.2.4.B. Fundamental matrix

For the most general case of a 3D non-deformable object moving in a 3D world, the set of point correspondences {X i ↔ X i '} in two perspective-projection images are related by a Fundamental matrix, such that:

𝑋′ 𝑖 𝑇 . 𝐹. 𝑋 𝑖 =0 / i=1…N (eq. III-4)
The Fundamental matrix F has Rank 2, and det(F)=0. It also has seven degrees of freedom, and it can map each point in an image to its corresponding point in the other image.

Several methods for estimating the Fundamental matrix have been studied. Some methods are linear, whereas others are not. A list of these methods would include sevenpoint algorithms, eight-point algorithm, methods based on minimizing the geometricreprojection error (with the so-called Gold Standard method), minimizing the first-order geometric error (i.e., the Sampson distance and the symmetric epipolar distance), Levenberg-Marquardt optimization, an iterative linearized method, and others. For more information, the reader is referred to the references: (R. [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF], (Quan-Tuan [START_REF] Luong | The fundamental matrix: Theory, algorithms, and stability analysis[END_REF] , and(Quang-Tuan Luong, Deriche, Faugeras, & [START_REF] Luong | On determining the fundamental matrix : analysis of different methods and experimental results[END_REF]. NB: Before estimating the fundamental matrix, we should mention that in real images, similarly to the note mentioned above, the position of the points x i and x′ i is perturbed by noise, and so, image measurement errors will occur in both the images, for that the epipolar constraint x′ i T . F. x i = 0/ i=1, …, N is not fully satisfied, then: However, rather than searching for the algebraic error ε, a geometric error can be often measured on image planes. This leads to the definition of epipolar distance (error), which is, in the right image, is the perpendicular distance from the point x′ i to the epipolar line l′ = F. x i , and is written as (x′ i, F. x i ) . In the same manner, in the left image it will be d(x i, F T . x′ i ).

𝑥′
In General, the epipolar distance is computed for both images to avoid any bias in any computation using the epipolar distance, and that is what's called Symmetric Epipolar Distance:

For a couple of corresponding points x i ↔ x′ i :

𝐸𝑟 𝑖 𝐹 = 𝑑(𝑥′ 𝑖, 𝐹. 𝑥 𝑖 ) 2 + 𝑑(𝑥 𝑖, 𝐹 𝑇 . 𝑥′ 𝑖 ) 2 (eq. III-6)
For all the corresponding points x i ↔ x′ i / 𝑖 = 1 … 𝑁 :

𝐸𝑟 𝑒𝑝 = ∑ (𝑑(𝑥′ 𝑖, 𝐹. 𝑥 𝑖 ) 2 + 𝑑(𝑥 𝑖, 𝐹 𝑇 . 𝑥′ 𝑖 ) 2 ) 𝑁 𝑖=1
(eq. III-7)

Also, other error measurements are being used, that can be found in the literature.

III.3. Proposed approach

In the real world, a general moving object has displacement, for example, from position A 3D to position B 3D . Its features correspond at both positions, as do the points along its surface. This displacement can be represented by 3D motion vectors { V ⃗ ⃗ i , 𝑖: 0 𝑡𝑜 𝑛 … }. In a video, using a general projective camera, this object is projected on image planes (of different positions A 2D , B 2D , C 2D …) where each image plane (frame) is the projection of a position in 3D space at different time. Subsequently, 3D motion vectors V ⃗ ⃗ i are projected to 2D motion vectors v ⃗ i from frame position A 2D to frame position B 2D , where each vector represents the displacement of a pixel from one image to another. This gives the corresponding points x i ↔ x i ', where x i and x i ' are the two extremities of the vector v ⃗ i .

The main questions here are: 1. First, how to detect object motion? 2. Then, how to find displacements of object points from frame position to another (2D motion vectors)? Answers can be found in the motion estimation sub-section III.3.1.

When a static camera is used, the backgrounds in the frames are static. Consequently, background estimated motion in the frames will be a null vector. Moreover, because there is only one moving object in the scene, the motion-estimation will point out the object movement represented by motion vectors. This process begins by deciding, for each temporal motion, whether the displacement between time t 1 and t 2 is deformable or not. In the case of non-deformable object motion, there will be a particular transformation to map 𝑥 𝑖 to its corresponding 𝑥′ 𝑖 . This leads us to epipolar geometry and the Fundamental matrix-and, in special cases, to the Homography matrix.

Later, we will attempt to calculate this transformation. If found with a correct mapping, the temporal displacement (motion) is classified as non-deformable, else, it is considered as deformable. However, in each of the above cases, the object can be either. Thus, we studied the temporal consistency of the displacements to determine whether the object is deformable or non-deformable.

In summary, first, we detect object movements and estimate the motions vectors in the scene, using the optical flow as a motion-estimation method (explained in detail in III.3.1,below). The output from this step will be motions vectors belonging to the moving object, false vectors detected outside the moving object, falsely estimated vectors inside the moving object, and unusable motion vectors. Then, we filter these motions vectors (as explained in III.3.2,below). This step removes false, wrongly estimated, and unusable motion vectors. Only the true positive vectors belonging to the moving object remain. Next, we search for the transformation (Fundamental/Homographic matrix), if there is one, which satisfies these movements (as discussed in Subsection III.3.3). The output from this is the estimated transformation H/F. Subsequently, we determine whether the transformation correctly maps the two sets of corresponding points. By reference to this, the decision is made about the detected temporal motion as to whether it is deformable (as detailed below, in III.3.4). Finally, from the sequence of the temporal deformability of movements, we can infer the deformability of the moving object (for which, see Subsection III.3.5). This step will ultimately classify the object moving through the scene as deformable or nondeformable. We explain the sequence of procedures and the proposed algorithms in Subsection III.3.6. Thus, in short, the proposed approach follows the five steps represented in Figure III-3:

III.3.1. Motion estimation

This study involves considering the projected object points on two image plans. The transformation, if any, that correctly maps the corresponding points must then be found. To this end, a reliable method is needed-one that can produce a very dense, accurate (to the extent of using sub-pixels), and regular field of vectors representing the pixel displacements of a moving object. Moreover, the capability to track each moving object pixel through frames is required. This must be combined with the ability to estimate any kind of movement, even slow movement or object rotation.

Many, mentioned in the related works, approaches for motion estimation (Optical flow approaches, feature-based approach, block matching…) were well examined and tested.

The most competitive methods, useful for this study, are the optical flow, the block matching and the feature correspondence. Conceptually, the optical flow field is a set of condensed feature matches, having one match for every image pixel. Conversely, one can view feature-correspondence methods as optical flow computation at a few selected locations, with a high probability that the optical flow will be correctly estimated. Two major differences between feature correspondences and optical flow may be identified [START_REF] Fakih | Structure from Motion: Combining features correspondences and optical flow[END_REF]:

-Feature correspondences have a higher signal to noise ratio.

-The number of reliable feature correspondences is lower than the number of optical flow values. The problems related to methods based on the optical flow, comprising the dense sub-pixel block matching method, are mainly the computation time and the unreliability when estimating fast (i.e., large) motion. The problems with a feature-correspondence method concern the non-dense (i.e., the lack of density) and non-periodic (i.e., irregular) field of motion vectors.

Considering the optical-flow method, the time-consumption problem can be improved by parallelization when applicable. Moreover, the reliability problem in estimating large motion can be solved by introducing a pyramidal implementation, which allows for faster (i.e., larger) motion tracking.

On the other hand, the lack of density and periodicity in the motion vectors that result from using feature-correspondence methods can be overcome by constructing a dense velocity field from a sparse-correspondence-point velocity field. However, this solution is considerably difficult to apply.

In addition, for further application and interpretation of an action, it is not possible to sacrifice the availability of dense and regular information in order to avoid missing any part of the object's body. In fact, missing such an articulation leads to a misclassification of the object. Thus, every articulation of the body must be segmented to obtain further information concerning the action. Working with a dense and regular field will be beneficial when checking the deformability to determine the percentage or degree of the object's deformability. Finally, a dense and regular field may help with segmenting objects in the scene.

To achieve our main goals with best results, when no prior information about the content of the scene is available-and with a minimum number of hypotheses, assumptions, and constraints-an optical-flow method is adopted. In fact, such a method better suits this type of study, in spite of the complexity and the computational time required. Furthermore, optical-flow methods and algorithms are widely parallelizable, and they can take advantage of advances in processing technology and parallelizing systems. Moreover, the optical-flow method uses a pyramidal implementation that allows for faster motion tracking. In addition, this method deals with some remaining problems listed.

The optical-flow approach is a well-known concept that has been exploited for several years, with many techniques and a variety of methods [START_REF] Barron | Performance of optical flow techniques[END_REF].

One of the most interesting methods of working with the optical flow is the Lucas-Kanade method [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF]. However, experiments with the Lucas-Kanade algorithm reveal that it is unsuitable for large displacements caused by the approximation when omitting higher-order terms (i.e., higher than the first terms in the optical-flow equation) [START_REF] Bruhn | Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods[END_REF], [START_REF] Wedel | Stereo Scene Flow for 3D Motion Analysis[END_REF]. Thus, improvements (e.g., the pyramidal approach [START_REF] Burt | The Laplacian Pyramid as a Compact Image Code[END_REF]) have been made to the Lucas-Kanade algorithm. Marzat [START_REF] Marzat | Estimation temps réel du Flot Optique[END_REF] presented a pyramidal implementation of the Lucas-Kanade method with regularized least squares1 . To ameliorate the results, Marzat used several optimization techniques: he implemented a pyramidal approach (i.e., a multiresolution approach) and in plus an iterative and temporal refinement. The reader is referred to [START_REF] Marzat | Estimation temps réel du Flot Optique[END_REF] and [START_REF] Dumortier | Perception monoculaire de l'environnement pour les systèmes de transport intelligents[END_REF], to read more about the pyramidal representation and its advantages after implementing Lucas-Kanade method, and Marzat's pyramidal method.

Marzat [START_REF] Marzat | Estimation temps réel du Flot Optique[END_REF] and [START_REF] Dumortier | Perception monoculaire de l'environnement pour les systèmes de transport intelligents[END_REF][START_REF] Dumortier | Perception monoculaire de l'environnement pour les systèmes de transport intelligents[END_REF] conducted many comparative tests, focusing on differential techniques (viz., the Lucas-Kanade algorithm, the Horn-Schunck algorithm, and block-matching approaches). This is related to the fact that other techniques do not appear dense, nor do they use excessive filtering or many parameters. According to Marzat, his algorithm is more accurate than the Lucas-Kanade and block-matching algorithms. On one hand, as we saw, the Lucas-Kanade algorithm is unsuitable for large disparities. On the other hand, a block-matching algorithm using typical techniques, such as those explained in [START_REF] Khammar | Evaluation of different block matching algorithms to motion estimation[END_REF], cannot give sub-pixel-wise information without further processing. Marzat solved these problems by using scale pyramids which reduces the image resolution. The disparities gained here are then used in the higher resolution images.

Moreover, we did several comparison tests with optical-flow algorithms to identify the method that best suits our type of work.

The corpus mentioned in Section III.4 was used. We compared Marzat's algorithm with the following algorithms, found in the Computer Vision System Toolbox in Matlab: the Lucas-Kanade algorithm, the Horn-Schunck algorithm, and block matching. We determined that the solution proposed by Marzat best suits better this type of studies. Indeed, the first two algorithms (viz., Lucas-Kanade and Horn-Schunck) are decidedly unsuitable for large disparities [START_REF] Horn | Determining optical flow[END_REF], (Meinhardt-Llopis, Sánchez Pérez, & Kondermann, 2013), [START_REF] Bradski | Learning OpenCV: Computer Vision with the OpenCV Library[END_REF], [START_REF] Michael | Robust incremental optical flow[END_REF], and the block-matching algorithm does not provide subpixel-wise information without further processing.

To summarize Marzat's algorithm: -It offers more precision (i.e., sub-pixel estimation) for the motion vectors -It does not require much filtering -It detects both slow and fast motion -It is more coherent and consistent -It is completely parallelizable Thus, Marzat's optical-flow method meets the objectives of the present study, and it was implemented in this work.

Estimating motion with Marzat's algorithm produces motion vectors belonging to the moving object, false vectors detected outside the moving object, falsely estimated vectors of the motion inside the moving object, and unusable motion vectors. An example is shown in Figure III-4. Hence estimating motion with Marzat's algorithm requires filtering to ameliorate its results and to remove unreliable motion vectors. This process is described in the following Subsection, III.3.2.

It is well known that the object-boundary motion might not always be consistent with the object's 3D motion. However, we consider this effect marginal, and it is already well filtered by Marzat's algorithm (with a smoothness effect as a result of the pyramidal approach). In addition, for the remaining unreliable object-boundary motion, we must use thresholds to determine the inconsistency that will be taken into consideration when classifying the object as deformable or non-deformable.

III.3.2. Motion filtering

The purpose of this step is to eliminate all unreliable motion vectors data that have not been already filtered by Marzat's algorithm. The false-positive appearance of vector movements in uniform areas is the first case that can be detected. The second case appears when the detected vectors are parallel to the local texture (see Figure , meaning that any estimation of those vectors will be erroneous. This is a typical limitation, and one that is common to all optical-flow estimation tools. The Lucas-Kanade algorithm can identify such cases when motion vectors are estimated with the help of a tensor-structure matrix of "weak rank 2" (i.e. when at least one of its eigenvalues is close to zero). In addition, especially small vectors are insignificant to further interpretation. Thus, and to avoid further critical errors in processing, it is indispensable to filter all such vectors, even if, in doing so, there is a risk of losing some true-positive vectors. For this type of work, it is better to have fewer reliable vectors than many that are unreliable. All the remaining vectors should belong to the moving object, and they should be reliable and regularly dispersed over the parts of the object. To achieve this, after detecting and estimating motion vectors with Marzat's algorithm for each pixel in the desired frame, three simple filters are used: the small-vectors filter, uniformity filter, and texture filter.

III.3.2.A. Small-vectors filter

All insignificant vectors with a very small abscissa and ordinate (<0.5) are eliminated. Indeed, these vectors are insignificant in searching for a transformation, and they can decrease the reliability of the calculated transformation. Moreover, those vectors are basically considered as unimportant motion (e.g., the motion of tree leaves), noise, or poor detections.

III.3.2.B. Uniformity filter

Where there are uniform areas (i.e., when the variation of the intensity is small) in the frame, Marzat's algorithm detects false-motion vectors. Thus, we check each of the remaining motion vectors in the desired frame after the application of the small-vectors filter, if a vector exists in a uniform area, it will be eliminated.

III.3.2.C. Texture Filter

Marzat's method uses the Lucas-Kanade approach, which is based on motion-vector estimation according to gradient calculations. That can generate false vectors estimation, especially on edges where vectors appear parallel with local texture. In this filtering, these vectors must be eliminated. Thus, for each of the motion vectors remaining after small vectors filtering and uniformity filtering, in desired frame, we find intensity's variations in the vector surrounding block, in the direction and orientation of this motion's vector. If the average of intensity's variations (differences) is low, so the motion's vector is on the same direction and orientation as the local texture; that means it is not reliable and it will be eliminated.

Applying the 3 filters on

III.3.3. Transformation

The output from the motion-filtering step can be considered as two sets of corresponding points. That is, it is the perspective projections from two positions taken with a general projective camera2 of a general moving object.

The final step in this study is to determine whether the object is deformable. It is necessary to identify deformable motion (i.e., displacements) among non-deformable motion. Thus, it is essential to begin with a kinematics theory of non-deformable bodies. For general 3D non-deformable static bodies, a well-known transformation exists between two corresponding features taken from two different camera positions at two different times. This case is equivalent to the case of one static camera taking two images of a 3D nondeformable moving body at two different times. Thus, when a non-deformable object is observed in two perspective-camera views, its feature correspondences satisfy an epipolar constraint for a general non-deformable body. The transformation that can map the correspondence of the points is called the Fundamental matrix. The study of the Fundamental matrix is part of epipolar geometry.

In order to find out whether the temporal displacement of the object is deformable, the deformability constraint of a non-deformable body motion will be identified. This can be done by finding a Fundamental matrix that is able to correctly map the set of points X i of the object from one image plan to its corresponding X' i in the other image. If this Fundamental matrix can be found, the displacement of the object is non-deformable; if not, then the displacement is deformable.

As stated before, when a non-deformable object is observed in two perspectivecamera views, its feature correspondences satisfy an epipolar constraint for a general nondeformable-body. However, it can also be satisfied by an planar projective transformation constraint (i.e., with 2D homography) in several cases, such as for planar objects (or objects that are assumed to be planar objects), certain objects in special cases (e.g., distant or small objects), planar motion (i.e., pure translation or rotation in the image plan when the rotation axis is an orthogonal image plan), or when using a 2D camera (also known as planar camera) (orthographic). In such cases it is clear that the planar motion is a degenerate nondeformable-body motion. The Homography matrix can be successful at replacing the Fundamental matrix in these cases, and it can be considered a simplification of the problem. It can be used for a lot of applications: camera fixed in mobile vehicle, mobile vehicle in urban or building environment and robots in movements.... The homography matrices were also tested, by a simple replacement of the fundamentals one.

With two sets of corresponding points, the Fundamental and Homography matrices can always be estimated with existing estimation methods. The problem reverts from finding a Fundamental (or Homography) matrix to determining whether the estimated Fundamental (or Homography) matrix correctly maps the corresponding points.

III.3.3.A. Homography (Projective transformation)

Various algorithms were proposed for homography estimation. In the present work, deformable motions and objects should be distinguished from non-deformable ones. To this aim, an estimation algorithm will be used to estimate the homography that relate the corresponding points of the objects, either if the object is deformable or is not-deformable; and results will be compared. Thus, a comparison method with the same estimating algorithm will be employed. This will attenuate the result errors effects.

Thus, in this study, we used the well-known DLT3 algorithm coupled with normalization4 by Hartley and Zisserman (R. [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF] for estimating the homography . The DLT estimates the Homography matrix, given two sets of corresponding points. The normalized DLT (NDLT) algorithm is a linear homogeneous solution based on minimizing a suitable cost function to numerically solve the linear equations of the Homography.

The solution proposed by the NDLT is the method of least squares using Singular Value Decomposition (SVD) [START_REF] Abdel-Aziz | Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry[END_REF].

Nb: As a code for NDLT, we used the Matlab function vgg_H_from_x_lin [START_REF] Zisserman | Three main categories of approaches for classifying moving objects[END_REF].

III.3.3.B. Fundamental Matrix

General projective camera, which is as uncalibrated camera, is studied in this work seeking more generality. Two perspective views (two images) are considered: right and left. Beside this, set of points in one image and its correspondences in the other are also represented. The aim in this part is to find the transformation that can map each of the points in one image to its correspondence, in the other image.

In this study, the normalized 8-point algorithm (N8PA) is used to estimate the Fundamental matrix, because it provides adequate results and because it is quick and easy to implement. The 8-point algorithm was first introduced by H. Christopher [START_REF] Longuet-Higgins | A computer algorithm for reconstructing a scene from two projections[END_REF], and then coupled with the normalization by Hartley and Zisserman (R. [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF].

The N8PA estimates the Fundamental matrix, given two sets of corresponding points. The starting equation is different from the one used with the homography but is solved using the same main steps (minimizing a suitable cost function to numerically solve the linear equations and least squares using Singular Value Decomposition) (R. [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF].

Nb: As a code for N8PA, we used the Matlab function fundmatrix (Kovesi, n.d.).

III.3.4. Deformable and Non-Deformable Motions

After calculating the optical flow between two frames using Marzat's algorithm-and after filtering the motion fields and estimating the Fundamental matrix F and the Homography matrix H that relate the corresponding points ((x i ↔ x i ' / i= 1… N) covering the object in the two image plans in all cases (i.e., for both deformable and non-deformable objects)-the motion deformability can be investigated. Through this investigation, it is possible to determine whether the projected displacement of the object from position A 3D to position B 3D (between times t 1 and t 2 ) is a deformable or non-deformable displacement.

The investigation uses the property of non-deformable body motion discussed above (in Subsection III.3.3). In the case of non-deformable motion, its feature correspondences can satisfy a Fundamental matrix (in general cases) or a Planar-Projective transformation (in special cases). Thus, F (or in special cases, H) can be checked. A correct mapping of the corresponding points means that the motion is non-deformable, according to the matrix F (or H) that is found. Otherwise, the motion is deformable. As a consequence, the problem now is to determine whether the transformation (F or H) is, in fact, a correct mapping of the corresponding points.

Ideally, the transformation F (or H) is a perfect mapping of the corresponding points, and F (or H) can correctly map all N points x i to x i ' and vice versa:

 For H: 𝑥 𝑖 ′ should coincide with 𝐻. x i for all the N points 𝑥 𝑖 (x′ i = 𝐻. x i / 𝑖 = 1 … 𝑁 ) and 𝑥 𝑖 should coincide with 𝐻 -1 . x′ i for all the N points 𝑥 𝑖 ′ (x i = 𝐻 -1 . x′ i / 𝑖 = 1 … 𝑁 ).
 For F: the case is a little bit different: 𝑥 𝑖 ′ should be on the epipolar line 𝑙 𝑖 ′ corresponding to 𝑥 𝑖 for all the N points 𝑥 𝑖 (𝑙 𝑖 ′ = 𝐹. 𝑥 𝑖 / 𝑖 = 1 … 𝑁 ) and 𝑥 𝑖 on the epipolar line 𝑙 𝑖 corresponding to 𝑥 𝑖 ′ for all the N points 𝑥 𝑖 ′ (𝑙 𝑖 = 𝐹 𝑇 . x′ i / 𝑖 = 1 … 𝑁 ).

However, because of real images errors, the position of the points x i and x i ' is disturbed by noise. Image-measurement errors will occur in both images and the estimation of F and H will not be perfectly accurate. In such cases, two issues are taken into consideration:

 The error margin when mapping the corresponding points (x i ↔ x i ' ): establishing the acceptable error in mapping x i to x i ' and vice versa. For that, we calculate the error for each two corresponding points (x i ↔ x i ' :

o For H: by the Symmetric Transfer Error, described in Subsection III.3.3, or by the mean distance (error): where γ H , γ' H and γ F , γ' F are the mapping error thresholds for H and F, respectively. γ H , γ' H and γ F , γ' F are later calculated in order to apply the thresholds to all types of objects and movements.

𝐸𝑟′ 𝑖 𝐻 = (𝑑(𝑥 ′ 𝑖 , 𝐻. 𝑥 𝑖 ) + 𝑑(𝑥 𝑖 , 𝐻 -1 . 𝑥 ′ 𝑖 )) 2 ⁄ (
 The error margin in the percentage of correctly mapped points: establishing the acceptable percentage of points that are not correctly mapped, even though both sets of corresponding points are considered to be correctly mapped, in general.

As mentioned above, an error will occur when estimating F or H in mapping the corresponding points. Thus, it is not the case that all the N points in the two sets of corresponding points will be correctly mapped with H or F. To consider the two sets of corresponding points x i ↔ x' i / i = 1…N as correctly mapped, it will be sufficient if the percentage of the number of correctly mapped points is more than a certain threshold: δ F = (N' F .100 / N) for F, or δ H = (N' H .100 / N) for H; with N' F <=N and N' H <=N are the number of correctly mapped points corresponding to F and H. Moreover, δ F and δ H should be generalized as much as possible so that they can be applied for all types of objects and movements. This must be accomplished in such a way that, whatever the object is, and for any temporal movement between the two frames im n-1 and im n , the two sets 2N of filtered and corresponding points can be found. Then, F (and, respectively, H) can be estimated; subsequently, the percentage of correctly mapped corresponding points (p F for F, and p H for H) can be found. Finally, if p F >= δ F (p H >= δ H ), then the two sets of corresponding points are correctly mapped by F or H, and the transformation represents a correct mapping of the two sets of corresponding points between im n-1 and im n . As a result, the temporal movement of the object is classified as non-deformable motion. Else, it is deformable. In other words, p F (p H ) can be seen as the non-deformability percentage of the temporal movement of an object for F (H); subsequently, δ F (δ H ) is the motion non-deformability threshold for F (H). The values δ F and δ H are calculated as per the method described in Section III.4.

Because δ F and δ H should be generalized as much as possible so that they can be applied to all types of objects (deformable, non-deformable, small, medium, and large, with texture, smooth, etc.) and movements (slow, medium, fast, small, large, in all directions, etc.), the F or H motion non-deformability thresholds δ F and δ H should be investigated as to whether they can be affected by the following two parameters:  Number of motion vectors: the number of couples among the corresponding points.

Several tests were conducted with different scenes and scenarios, by taking several sets of random vectors in detected objects to determine whether the number of motion vectors needed to calculate the Fundamental F or the Homography H can seriously affect δ F and δ H . Moreover, the correlation was calculated.  Average Length of the Motion Field (ALMF): the average motion-vector length. Several tests were conducted by taking the same moving object and similar motion with different vector lengths, resulting in different average lengths. Based on tests, it was clear that the number of motions vectors does not seriously affect the motion non-deformability thresholds δ F and δ H for F and H, respectively. This conclusion is evident because only a few vectors (four vectors for H, and eight vectors for F) are needed to define and represent the true temporal displacement of the object. Thus, the density of the motion field can be reduced in order to diminish the time required for filtering. This can be done without losing the regular dispersity needed to cover the entire object, and thus without losing any important information about the object.

Concerning the length of the motion, initially in the experiments, the mapping error threshold for H (respectively, F) is fixed regardless of the motion length, and different lengths of motion can significantly affect δ F and δ H in such a way that the smallest average length of the motion field will have the highest motion non-deformability threshold for H (respectively, F) (see Section III.4) to maximize the results (i.e., to minimize the errors in discriminating between non-deformable and deformable).

Therefore, as the generality is pertinent to solving the problem of deformable and non-deformable motion with different types of videos, different objects, and objects moving at different speeds with different motion-field lengths. Thus, a normalization step is added to normalize the length of the motion field after motion filtering and before calculating the transformations. For that, all motion vectors are normalized to an average motion-field length equal to n (n=1, 2, 3, 4 … round (original average length)).

The Fundamental F n and the Homography H n were calculated for each of normalized set of vectors corresponding to normalization level n. Therefore, instead of finding the motion non-deformability threshold δ F for F (and, respectively, δ H for H), a set of thresholds δ F n (respectively, δ H n ) must be found: one for each normalization level n:

δ F n = {δ F 1 , δ F 2 , δ F 3 … δ F i … δ F n }(respectively, δ H n = {δ H 1 , δ H 2 , δ H 3 … δ H i … δ H n }).
N.B.: δ F n and δ H n are used with the Symmetric Epipolar Distance and the Symmetric Transfer Error, respectively, but when using the mean distance, we must find δ'

F n = {δ' F 1 , δ' F 2 , δ' F 3 … δ' F i … δ' F n }(respectively, δ' H n = {δ' H 1 , δ' H 2 , δ' H 3 … δ' H i … δ' H n }).
In summary, in seeking to maximize the study's results by finding the ultimate motion non-deformability thresholds for discriminating deformable and non-deformable motion, two essential parameters had to be considered:  The normalization level. It should be noted here that, for small object movement, deformable motion can be confused with non-deformable motion in the real world. Furthermore, the length of the motion vectors and the difference in length among motion vectors are very small. Thus, the Fundamental matrix F, the Homography H, and the motion non-deformability thresholds are unreliable, which will raise the percentage of errors when discriminating between deformable and non-deformable motion. Moreover, by having especially long movement vectors, errors in estimating the motion vectors (with the optical flow) and in estimating F and H will be duplicates, and the motion non-deformability thresholds will be unreliable, which will again increase the percentage of errors.

Following the experiments, and in order to obtain reliable results, the Average Length of Motions Field ALMF should fall between seven and ten. For that, the motion vectors inputted during the third step (viz., transformation) should have an average length of between seven and ten. By changing (i.e., by eloigning or approaching) the input-compared frame im i (i.e., the frame compared with the current frame im n ) and repeating (i.e. reiterating) the first and the second steps (viz., motion estimation and motion filtering), the desired average length of the motion field can be obtained. For example: instead of comparing frame X n with frame X n-1 , we can compare it with frame X n-2 or X n-3 , or it can be compared with another frame until a suitable result is found.

Consequently, after finding the filtered motion-vectors field, the field is normalized, and for each normalization n, the Fundamental matrix F n is estimated (and, respectively,

III.3.5. Deformable and Non-Deformable Objects

Until now, object motion has been classified independently for each frame (i.e., according to the movement or displacement in the frame), but frame-by-frame detection results in many classification errors. Moreover, whether the temporal displacement is deformable or non-deformable does not necessarily indicate that all its motion will be of that sort. It is not the case that objects deformability can be inferred from a single motion exclusively. For these reasons, the entire series of the object's apparent motion should be considered.

When an object appears in frames, the series of its temporal motion (Motions5 : X i , X i+1 , X i+2 ... X j , ... X n ) will be studied and classified as deformable or non-deformable motion (like 𝐷 𝑖 , 𝐷 𝑖+1 , 𝑁𝐷 𝑖+2 ⋯ , 𝐷 𝑗 ⋯ , 𝑁𝐷 𝑛 ).

Two criteria should be taken into consideration:

 Errors in classifying the temporal motion: the temporal motion can be misclassified owing to errors in the motion-estimation algorithm and transformation-estimation errors caused by image noise, etc.  A deformable object can have non-deformable motion: A deformable object can be mistaken for a non-deformable object if it acts (i.e., appears) as a non-deformable object for an extended period. This occurs when the articulations of a deformable object are hidden at the time of motion or have the same displacements as an entire body, A good example could be when only the upper-body of a person (deformable object) appears and his hands moves as the same way as his upper-body (he's moving as one block). This can happen in one frame, or it can be consistent (e.g., a deformable object can be in motion for a long time with its articulations hidden); but this does not mean that the object is non-deformable. It should be noted that the opposite case is never true, because non-deformable objects cannot have deformable motion.

As a result, several cases must be considered:  In the series of motion classifications, if a motion is classified as deformable motion, then-if this is said of a deformable object-the classification is correct and should be left unchanged (i.e., it should not be corrected), but if deformable motion is said of a non-deformable object, then the classification is an error and should be corrected to non-deformable motion.  In the series of motion classifications, if a motion is classified as non-deformable motion, then-if this is said of a non-deformable object-the classification is correct and should left unchanged (i.e., it should not be corrected), but if non-deformable motion is said of a deformable object, then two cases appear: o Either, the classification is an error and should be corrected to deformable. o Or, the classification is correct and:  Either, the motion classification should be left unchanged, so that it remains nondeformable motion, taking into consideration that this is a case of consistent non-deformable motion from a deformable object.  Or, the motion classification should be changed to deformable motion, even though this is known to be untrue from a temporal point of view. It is, however, true from a general point of view for classifying the object per se as deformable, because this is the case of inconsistent non-deformable motion (in a one or two frames) of a deformable object.

Therefore, for all of these reasons:  At first, the temporal consistency will be studied when the motion is classified for all series of movements to correct classification errors, and to exclude the inconsistent nondeformable movements in a deformable object.  Second, object deformability will be inferred from the corrected (persistent) series of motion classifications, taking into consideration the consistent non-deformable motions of a deformable object.

The temporal information can be used to study the consistency of the motionclassification results and will lead to an improvement in the reliability of decisions over time.

In this study, we used the temporal-consistency algorithm proposed by Jaffré and Joly [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF]. Their goal was to improve the results obtained for an object detector operating independently on each frame of a video document; the results for the object detector are "smoothed" along the time dimension using a temporal window.

In order to reduce false detections (i.e., false alarms and incorrect detections), Jaffré and Joly [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF] propose exploiting the persistence properties of objects in a video sequence. They consider a temporal window of N frames centered on the subject frame. For this given frame, they count the number of occurrences of each object in the previous N/2 and the subsequent N/2 frames. Then, only the objects whose number of appearances is above a threshold (N 2 ) are validated. A probabilistic approach is used for a theoretical computation of the thresholds N and N 2 .

To validate all the correct detections, and reject all the false alarms, they search N and N 2 so as to maximize:

Having X =Number of correct detections in N frames / Y =Number of false alarms in N frames. 𝑝 𝑑 is the probability of success/ 𝑞 𝑑 = 1 -𝑝 𝑑 is the probability of failure. 𝑝 𝑓 is the probability to have a false alarm in a frame/ 𝑞 𝑓 = 1 -𝑝 𝑓 . Or they proposed to find N and N 2 by maximizing recall and precision: But they are lead to the same results of the first equation. They proposed a numerical resolution for the maximization of the two expressions.

arg max 𝛼 × 𝑟𝑒𝑐𝑎𝑙𝑙 + (1 -𝛼) × 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = arg max 𝑁,𝑁 2 𝛼𝑃(𝑋 ≥ 𝑁 2 ) + (1 -𝛼) 𝑃(𝑋 ≥ 𝑁 2 ) 1 + 𝑃(𝑋 ≥ 𝑁 2 ) -𝑃(𝑌 < 𝑁 2 ) (eq. III-11) arg max 𝑁,𝑁 2 𝑃[(𝑋 ≥ 𝑁 2 ) ∩ (𝑌 < 𝑁 2 )] = arg max 𝑁,𝑁 2 𝑃(𝑋 ≥ 𝑁 2 ) 𝑃(𝑌 < 𝑁 2 ) = arg max 𝑁,𝑁 2 ( ∑ 𝑃(𝑋 = 𝑖) 𝑁 𝑖=𝑁 2 ) ( ∑ 𝑃(𝑌 = 𝑖) 𝑁 2 -1 𝑖=0 ) = arg max 𝑁,𝑁 2 ( ∑ 𝐶 𝑁 𝑖 𝑝 𝑑 𝑖 𝑞 𝑑 𝑁-𝑖 𝑁 𝑖=𝑁 2 ) ( ∑ 𝐶 𝑁 𝑖 𝑝 𝑓 𝑖 𝑞 𝑓 𝑁-𝑖 𝑁 2 -1 𝑖=0 ) (eq.
The aim in our study, however, is not object detection or face localization in a shot, but rather classification. Strictly speaking, then, in our study there are no misdetections, but only correct or false classifications. Misclassifications (i.e., false classifications) will merely be a failure in the classification. As a result, the algorithm in this work was subject to a few modifications: e.g., X and Y used in the maximization equation are assumed to be independent in [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF], which is not the case in our study. Therefore the equation becomes:

(eq. where X is the number of correct classifications in N frames, and Y is the number of false classifications in N frames. In [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF], 𝑝 𝑑 is the probability of success, 𝑞 𝑑 = 1 -𝑝 𝑑 is the probability of failure, 𝑝 𝑓 is the probability of a false alarm in a frame, and 𝑞 𝑓 = 1 -𝑝 𝑓 ; in our case (without misdetection), 𝑝 𝑓 = 𝑞 𝑑 = 𝑞, and 𝑞 𝑓 = 𝑝 𝑑 = 𝑝. NB: while the problem has been change to a classification, and, X and Y are not completely independent, in addition, Y=N-X, it seems enough to maximize only the X term without the Y term (arg max 𝑁,𝑁 2 𝑃[(𝑋 ≥ 𝑁 2 ) ]). But the need of both of the terms (X and Y) in the maximization equation is proved in the Appendix VIII.3. Also, as reminder, P(A/B) it's the conditional probability, sometimes denoted P B (A) , it's the probability of A knowing that B have occurred.

Now, 𝑃(𝑋 ≥ 𝑁 2 ) = ∑ 𝐶 𝑁 𝑖 𝑝 𝑖 𝑞 𝑁-𝑖 𝑁 𝑖=𝑁 2
and 𝑃((𝑌 < 𝑁 2 )/(𝑌 ≤ 𝑁 -𝑁 2 )) will generate two cases:

(N 2 <= N-N 2 => N 2 <=N/2) or (N 2 >N-N 2 => N 2 >N/2):  If N 2 <=N/2, then 𝑃((𝑌 < 𝑁 2 )/(𝑌 ≤ 𝑁 -𝑁 2 )) = ∑ 𝐶 𝑁 𝑖 𝑞 𝑖 𝑝 𝑁-𝑖 𝑁 2 -1 𝑖=0 . Figure III-12: The case N 2 <=N/2.  If N 2 >N/2, then 𝑃((𝑌 < 𝑁 2 )/(𝑌 ≤ 𝑁 -𝑁 2 )) = ∑ 𝐶 𝑁 𝑖 𝑞 𝑖 𝑝 𝑁-𝑖 𝑁-𝑁 2 𝑖=0
, because the 𝑃((𝑌 < 𝑁 2 )/(𝑌 ≤ 𝑁 -𝑁 2 )) when N-N 2 <Y<N 2 is equal to 0.

𝐴𝑟𝑔 𝑚𝑎𝑥 𝑁,𝑁 2 𝑃[(𝑌 < 𝑁 2 ) ∩ (𝑋 ≥ 𝑁 2 ) ] = 𝑎𝑟𝑔 𝑚𝑎𝑥 𝑁,𝑁 2 𝑃(𝑋 ≥ 𝑁 2 ) 𝑃((𝑌 < 𝑁 2 )/(𝑋 ≥ 𝑁2)) = 𝑎𝑟𝑔 𝑚𝑎𝑥 𝑁,𝑁 2 𝑃(𝑋 ≥ 𝑁 2 ) 𝑃((𝑌 < 𝑁 2 )/(𝑁 -𝑌 ≥ 𝑁2)) = 𝑎𝑟𝑔 𝑚𝑎𝑥 𝑁,𝑁 2 𝑃(𝑋 ≥ 𝑁 2 ) 𝑃((𝑌 < 𝑁 2 )/(𝑌 ≤ 𝑁 -𝑁2))
Finally, the maximization equation will be:

To find the optimal values for N and N 2 that suit our aim, the numerical resolution proposed in [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF] was used to maximize the expression, having a probability of success p = 82.58 and a probability of failure q= 100-82.58= 17.42. We selected in Table III-1 the case of the Fundamental matrix as the transformation, the Symmetric Epipolar Distance as the distance measure, normalization level two, the mapping error threshold 𝛾 𝐹 2 =2.2, and the motion non-deformability threshold 𝛿 𝐹 2 = 80.16.

However, the set of solutions was a plateau (see Appendix VIII.3), and a solution was found that can be generic to several applications. Thus, the couple (N, N 2 ) taken is: (N, N 2 ) = (11,[START_REF] Klappstein | Moving Object Segmentation Using Optical Flow and Depth Information[END_REF], where, 𝑃[(𝑌 < 𝑁 2 ) ∩ (𝑋 ≥ 𝑁 2 ) ] is maximized to 0.988454).

Then, to study the temporal consistency of a series of motion classifications, for each classification of motion deformability, we consider a window of 11 classifications (5 to the left, from before; 1 under consideration; and 5 to the right, from after). There are then four possible cases. If the desired classification (i.e., the classification result, under consideration, that we are studying in terms of its consistency) is deformable (or, respectively, non-deformable), the number of deformable-motion classifications (Nb) between the 11 motion classifications is counted: >= 6, the classification remains deformable (respectively, non-deformable).  If nb < 6, the classification should be changed to non-deformable (respectively, deformable).

 If nb
When this temporal-consistency algorithm was applied once to the corpus (on 24 series from different videos), the percentage of success for the entire algorithm increased by more than 6%, resulting in an 89% accuracy rate (see the examples in Table III-2, below). The outputs from the temporal-consistency algorithm are the corrected and smoothed series of motion classifications. However, in case there are still isolated classifications after applying the temporal-consistency algorithm, this algorithm can be reiterated as needed, until the final output is completely smooth and unchangeable (i.e., stable). When temporal-consistency was applied, it increased the percentage of success by more than 6%, see the examples in Table III ) / If N2 > N/2 (eq. taking into account working with that big variety of videos and those very hard scenes taken (critical scenes, see section III.4), and knowing that the 8.2% of errors were the errors of Marzat's algorithm cumulated with the errors of filtering, the errors of estimating the transformation and the errors of classification of motion deformability, this percentage is considered as a good percentage relatively.

Nb: Supposing that a 3D object seen by a camera can be considered a planar object, the Homography matrix can be used, but in such cases, the percentage of success will be 75% without temporal consistency, and 80% with temporal consistency.

The final step in classifying the object is simple. We classify the object as deformable or not by looking on the motion-classification series of its appearance. If all the persistent motion classifications are non-deformable, then the object is non-deformable. Yet, the existence of one sub-series of deformable classifications is sufficient for the object to be classified as deformable.

III.3.6. Proposed Algorithms

Having provided a thorough explanation of each step, we turn now to the process sequences used to determine deformable and non-deformable motions and objects, as detailed in the following algorithms:

Deformable and non-deformable motion algorithm

Step 1: Estimate motion between two frames using Marzat's algorithm (Subsection III.3.1).

This generates a motion field.

Step 2: Filter the motion field using three filters (Subsection III.3.2).

Step 3: If the average length of the motion field (ALMF) is not between 7 and 10 (Subsection III.3.4), Steps 1 and 2 should be repeated after changing (by eloigning or approaching) the input frame that is being compared with the current frame until the average length of the motion field falls between 7 and 10.

Deformable and non-deformable object algorithm (Subsection III.3.5)

Step 1: Apply the temporal-consistency algorithm to the series object-motion classifications.

This results in the application of the deformable and non-deformable motion algorithm to all moving objects appearing in the scene.

Step 2: If the smoothed motion-classification series results from Step 1 are not totally smooth, then Step 1 is repeated on this new series, until we obtain a final smoothed and unchangeable (i.e., stable) series.

Step 3: If a sub-series of deformable motion exists in the final smoothed and stable motionclassification series, then the object is deformable. If not, the object is nondeformable.

III.4. Experiments

This section describes the experiments conducted in order to determine the thresholds (viz., the mapping-error threshold and the motion non-deformability threshold) for each normalization level, distance type, and transformation type. Thresholds are essential for discriminating between deformable and non-deformable motion, and they affect the percentage of success corresponding to each case.

Because we could not find any real public dataset particularly dedicated to the study of deformable and non-deformable object classification with which to compare our proposed method, we created our own dataset [START_REF] Youssef | Deformable vs Non-deformable Dataset[END_REF]. By filming some videos, and collecting others from real video-surveillance cameras. In addition, we did not find, in any related research mentioned in Section III.2, any source code that could be used to test our dataset for the purpose of comparison. We should also mention that, because many of the videos used in our experiments were taken from real police video-surveillance cameras, they were not diffusible.

We tested our approach on 75 colour videos containing 30 different scene types with more than 2,100 tested frames. All videos were taken using a static camera.

A large variety of scenes was taken into consideration: -Many types of scenes.

-Deformable and non-deformable objects.

-Different resolutions.

-Different kind of actions (running, fighting, rolling, crashing…).

-Different speed of action (slow or fast).

-Different luminosities (indoor and outdoor).

-Different distances (close and far scenes).

The objects in these videos had diverse properties. They differed in: -Nature: there were adults, children, cars, doors, chairs, maps, boxes, bicycles, cats, and dogs. -Size relative to the screen: from small to large.

-Distance from the camera: from close to far.

-Depth: there were 2D-like objects (e.g., maps) and 3D objects (e.g., cars).

-Motion speed: from slow (e.g., a person walking) to very fast (e.g., a vehicle on the highway). -Nature of movement: translation, rotation, forward/backward, and skewed.

-Lighting conditions: from low lighting to balanced and well-contrasted scenes.

Thus, this dataset is considerably diverse, and this makes it ideal for our purposes, insofar as the motion in these videos is especially difficult to classify. Here, in C. Scene from "Bomb2": a deformable object with a non-deformable motion

Thresholds for the motion-deformability experiments

The thresholds used for deformable and non-deformable motion discrimination must be generic. Therefore, experiments were done on several different objects (i.e., both deformable and non-deformable objects). On a large variety of videos from the corpus mentioned above, and a large number of different objects (deformable and non-deformable) were chosen for tests.

For each of those objects, all the series of its temporal motions (Motions: X i , X i+1 , X i+2 ... X j , ... X m ) were studied when this object appears in m frames. For each frame and its corresponding one, the optical flow was calculated using Marzat's algorithm at first, after that the motion fields was filtered, then motion vectors were normalized, and for each normalization n (from 1 to round (ALMF) or 10), transformations (F n / H n ) that relates normalized corresponding points (𝑥̅ 𝑖 𝑛 ↔ 𝑥̅ ′ 𝑖 𝑛 / i= 1… p (p is points number) ) covering the object in the two image plans were estimated, (Fundamental matrix F by the Normalized 8-Point Algorithm (N8PA) and the Homography matrix H by the Normalized Direct Linear Transform (NDLT)).

For the Normalized 8-Point Algorithm (N8PA), among several possibilities, we tested 2 functions, the first was the Matlab library function estimateFundamentalMatrix, and the second was the Matlab function fundmatrix (Kovesi, n.d.). Results were close, but we preferred to work with fundmatrix [START_REF] Zisserman | Three main categories of approaches for classifying moving objects[END_REF].

For the Normalized Direct Linear Transform (NDLT)), 2 functions were tested, the first was the Matlab function vgg_H_from_x_lin [START_REF] Zisserman | Three main categories of approaches for classifying moving objects[END_REF], the second was the function homest2d of Sasikanth [START_REF] Sasikanth | 2D -2D Projective Homography (3x3) Estimation -File Exchange -MATLAB Central[END_REF]. Results were close, but we preferred to work with vgg_H_from_x_lin.

After obtaining the normalized corresponding points and the corresponding transformations (F n / H n ) for each normalization level, two thresholds are needed:  The mapping error thresholds corresponding to each normalization level n (𝛾 𝐹 𝑛 , 𝛾′ 𝐹 𝑛 for F n and 𝛾 𝐻 𝑛 , 𝛾′ 𝐻 𝑛 for H n ) for both distance types. The thresholds must be general for all videos, scenes, objects, and motions. These same thresholds are then used to calculate the percentage of correctly mapped points (𝑝 𝐹 𝑛 for F n , and 𝑝 𝐻 𝑛 for H n ).

 The motion non-deformability thresholds for each normalization level n (𝛿 𝐹 𝑛 , 𝛿′ 𝐹 𝑛 for F n , and 𝛿 𝐻 𝑛 , 𝛿′ 𝐻 𝑛 for H n ) for both distance types. Again, the thresholds must be general. These thresholds are then used to classify any object motion as deformable or not. We then calculate the percentage of success for any type of transformation (F or H), with any type of distance, for each mapping error threshold, and for each motion nondeformability threshold.

In the following subsection, for better understanding, we explain how to derive the motion non-deformability thresholds when the mapping error thresholds are fixed (i.e., when they are equal to one, for example). Then thresholds are improved when we apply the ultimate thresholds with variable mapping error thresholds.

A-Mapping Error Thresholds Fixed to 1:

Let the mapping error thresholds 𝛾 𝐹 𝑛 = 𝛾′ 𝐹 𝑛 = 𝛾 𝐻 𝑛 = 𝛾′ 𝐻 𝑛 = 1, where n is the normalization level (n =1…10). For each object motion in the scene, the percentage of correctly mapped points (𝑝 𝐹 𝑛 , 𝑝′ 𝐹 𝑛 for F, and 𝑝 𝐻 𝑛 , 𝑝′ 𝐻 𝑛 for H) is calculated. Let m be the number of motions tested for any given object appearance in the scene, and let 𝑀 𝑚 denote the set of all these motions: 𝑀 𝑚 = {𝑋 1 , . . . 𝑋 𝑖 … 𝑋 𝑚 }.

For each 𝑋 𝑖 we have 𝑝 𝐹 𝑛 , 𝑝′ 𝐹 𝑛 , 𝑝 𝐻 𝑛 , and 𝑝′ 𝐻 𝑛 for each normalization level n, and these are presented as 𝑝 𝐹 𝑛 𝑖, 𝑝′ 𝐹 𝑛 𝑖, 𝑝 𝐻 𝑛 𝑖, and p′ H n i. The set 𝑀 𝑚 contains deformable and nondeformable motion. Each motion is classified manually, as deformable or not, by reference to its movement between the two corresponding frames, and in a critical and rigorous way. For example, if only a small part of a human body (e.g., a part of a hand) is moving in a manner different from the body, regardless of whether this motion was correctly estimated with the Marzat optical flow, the object is considered deformable.

Let 𝐷 𝑟 denote the sub-set of the set 𝑀 𝑚 , with all the manually classified deformable motions. Similarly, sub-set 𝑁𝐷 𝑠 contains all the manually classified non-deformable motions, where r and s are the cardinalities for 𝐷 𝑟 and 𝑁𝐷 𝑠 , respectively, such that 𝑟 + 𝑠 = 𝑚. ) are found, they can be used for any kind of video, object, or motion.

However, when working with real images, we cannot find the motion nondeformability thresholds that completely split the two sub-sets 𝐷 𝑟 and 𝑁𝐷 𝑠 . Therefore, we settle for thresholds that conform to the following two conditions concurrently (see Figure :  The biggest number (or percentage) of motions in 𝑁𝐷 𝑠 have their own percentages of correctly mapped points (𝑝 𝐹 𝑛 𝑖, 𝑝′ 𝐹 𝑛 𝑖, 𝑝 𝐻 𝑛 𝑖 or 𝑝′ 𝐻 𝑛 𝑖), above the corresponding threshold.

Likewise, the minimum number of motions in 𝑁𝐷 𝑠 have their own percentages of correctly mapped points, below this same corresponding threshold. In other words, we retain the "best maximum" of non-deformable motions above the threshold, and an "acceptable minimum" of non-deformable motions below the threshold.  The biggest number (or percentage) of motions in 𝐷 𝑟 have their own percentages of correctly mapped points (𝑝 𝐹 𝑛 𝑖, 𝑝′ 𝐹 𝑛 𝑖, 𝑝 𝐻 𝑛 𝑖 or 𝑝′ 𝐻 𝑛 𝑖), below the corresponding threshold. Likewise, the minimum number of motions in 𝐷 𝑟 have their own percentages of correctly mapped points, above the corresponding threshold. In other words, we keep the "best maximum" of deformable motions below the threshold, and an "acceptable minimum" of deformable motions above the threshold.

Figure III-17 shows the ideal threshold (the yellow vertical line), where all non-deformable motions (in ND s ) have their respective percentages of correctly mapped points above the threshold, and all deformable motions (in D r ) have their respective percentages of correctly mapped points below this threshold. While for the discovered threshold (the violet oblique line), the maximum non-deformable motions (in ND s ) have their percentages of correctly mapped points above the threshold, and the maximum deformable motions (in D r ) have their percentages of correctly mapped points below this threshold.

Sub-set ND s of non-deformable motions

Sub-set D r of deformable motions For this purpose, a new kind of graph is used. We call it the "Best Maximum -Acceptable Minimum Graph". For any given normalization level n, transformation, and distance type, a graph for the motion non-deformability threshold (𝛿 𝐹 𝑛 , 𝛿′ 𝐹 𝑛 , 𝛿 𝐻 𝑛 and 𝛿′ 𝐻 𝑛 ) is obtained as follows:  The sub-set ND s of the non-deformable motions is sorted in descending order, according to the percentage of correctly mapped points for each frame in the sub-set.  On the other hand, the sub-set D r of the deformable motions is sorted in ascending order, according to the percentage of correctly mapped points for each frame in the sub-set.  A percentage is given for each element in the two sub-sets, representing its placement within the sub-set. This value is called the "Placement Percentage." For example, the 5 th element in ND s will be given the percentage Note that for any point (x, y) on the curve:  The abscissa (x) represents the Placement Percentage of this point according to its own series. For example, if x = 60 on Series 1, 60% of the non-deformable motions are above this point. If x = 60 on Series 0, however, 60% of the deformable motions are below it.  The ordinate (y) represents the percentage of correctly mapped points for this motion.

Thus, if y = 60, 60% of the point correspondences in the motion field are correctly mapped, given that the Fundamental F 2 is found.

Here the requested threshold t is a value of y=t, where:

((maximum of points in Series 1 are above line y=t) ⋂ (maximum of points in Series 0 are below line y=t)).

With this type of graph, the intersection of the two curves represents the best existing solution, where the maximum number of non-deformable motions (in ND s ) have their percentages of correctly mapped points above this coincidence point, and the maximum number of deformable motions (in D r ) have their percentages of correctly mapped points below this coincidence point. Let I(s,t) be the intersection point, with t denoting the requested threshold.

For example, in Figure III-18, if we settle for y= 70, rather than the intersection point, we know that 97% of non-deformable motions are above this threshold, and consequently well classified. However, only 62.6% of deformable motions are below this threshold, meaning that only 62.6% are well classified. Alternatively, if we take y=t=79.13 (the ordinate of the intersection point), then 82.16% of deformable and 83% of non-deformable motions are well classified, and this is the optimal percentage. Let I(s,t) be the intersection point, with t denoting the requested threshold. Notice that the abscissa, s, for the point of intersection I(s,t) represents, in this case, the percentage of success for the entire algorithm, insofar as the number of deformable motions and the number of non-deformable motions that are tested are approximately the same. Moreover, the Placement Percentage is the same for both series ND and D.

In summary, the intersection points in the graph indicate the best threshold for their normalization level, according to F or H with the usable distance. Accordingly, we calculate the motion non-deformability thresholds 𝛿′ 𝐹 𝑛 and 𝛿′ 𝐻 𝑛 for each normalization level (see following graphs for n=1..3). The temporal motion non-deformability thresholds for each normalization when the mapping error threshold is fixed to 1 (γ′ F n = 1 and γ′ H n = 1) are summarized in the Table III-3.

B-Ultimate Thresholds

As explained in the previous subsection, for each normalization level n, the intersection points along the curve represent the best motion non-deformability thresholds when the mapping error thresholds are fixed to 1. However, when the mapping error thresholds are variable, the percentage of correctly mapped points changes for each motion in 𝑀 𝑚 ( 𝑁𝐷 𝑠 , 𝐷 𝑟 ). Moreover, for all motion, the curves of variations change along with the intersection points.

For a given normalization level n, a transformation type, and a distance type, each mapping error threshold value leads to a different graph, and thus to a new intersection point. The best intersection point is the one with the highest abscissa (i.e., the highest percentage of success).

The aim is to find the mapping error threshold 𝛾 that can give the ultimate motion non-deformability threshold that maximizes the percentage of success. When the mapping error thresholds are variable, the search for the ultimate motion non-deformability thresholds can be done by finding the best intersection points by reference to the "Best Maximum -Acceptable Minimum" graph. To do this, we studied the variation of the curves' intersection points, according to the variation in the mapping error thresholds (𝛾 𝐹 𝑛 , 𝛾′ 𝐹 𝑛 for F n and 𝛾 𝐻 𝑛 , 𝛾′ 𝐻 𝑛 for H n ).

For each normalization level n, each type of distance measure, and for F and H, we generated a graph of the variation in the curves with intersection points according to the variations in the mapping error thresholds. For example, for the transformation F, normalization 3, the mean distance, and a mapping error threshold of 𝛾′ 𝐹 3 , varying between 0.2 and 10 with intervals of 0.2 (𝛾′ 𝐹 3 = 0.2: 0.2: 10), the graph will take the following form: In Figure III-25, it is clear that the mapping error threshold 𝛾′ 𝐹 3 = 1.4 is the threshold that maximizes the percentage of success to 82%, which corresponds to the ultimate motion non-deformability threshold of 76.92 %. Furthermore, for each normalization level n, we calculate the ultimate corresponding couple (mapping error threshold and motion nondeformability threshold) that maximizes the percentage of success for F n and H n , using different distance measurements. The values from this calculation are found in Table III-1. The thresholds in Table 3, and the ultimate thresholds in Table III-1, confirm that when the mapping error threshold is fixed, the best motion non-deformability thresholds will have decreasing values proportional to the normalization level (see Table III-3). However, if the mapping error threshold is variable in the appropriate way, the ultimate motion nondeformability thresholds will have approximately the same value, regardless of the normalization level or the distance type used (see Table III-1). This ensures high stability and reliability with regard to our algorithm.

On the basis of our experiments we recommend using the Fundamental matrix as the transformation, the Symmetric Epipolar Distance, normalization level 2, the mapping error threshold 𝛾 𝐹 2 =2.2, and the motion non-deformability threshold 𝛿 𝐹 2 = 80. [START_REF] Szegedy | Going Deeper With Convolutions[END_REF] (note: this is not to suggest that other thresholds are undesirable). As an example, when we compared our results for the two scenes above (viz., "Highway 2" where 𝑝 𝐹 2 = 94.1722, "Walking" where 𝑝 𝐹 2 = 68.7671, "Bomb2" where 𝑝 𝐹 2 = 95.2681, and "Big Map 3" where 𝑝 𝐹 2 = 98.4701) with the corresponding threshold (𝛿 𝐹 2 = 80.16), we can easily infer the deformability of each corresponding motion. The scene from "Highway 2" was classified as non-deformable motion, the scene from "Walking" was classified as deformable motion, the scene from "Bomb2" was classified as non-deformable motion, and the scene from "Big Map 3" was classified as non-deformable motion -and all classifications were correct.

Several different examples are described in Appendix VIII.3.

III.5. Conclusion

In this chapter, we proposed a threshold-based decision-making system aimed at determining whether an object or its motion corresponds to a non-deformable model using geometric projection modelling. The method relies on estimating parameters of a standard geometric transformation, which could be considered as a model of non-deformable object motion. The accuracy of this transformation in representing the object motion is then analysed to infer the actual deformability (or non-deformability) of the object.

We improved the results using temporal consistency, reaching a relatively high rate of precision (approximately 92%). Such a precision rate is largely sufficient to address new topics where knowledge about object deformability is an input.

This study provides the video surveillance research a rigorous and precise algorithm, which can be major feature when classifying the interaction between scenes objects. Also it is an important characteristic for the objects to be described at the final textual output.

IV. Description of video surveillance scenes

IV.1. Introduction

Most existing video surveillance systems provide the infrastructure to mainly capture video images, transmit, store, and distribute them, while the task of threat detection and analysis is left to human operators. Detecting an incident in a live source or searching the video archives for a specific one almost completely relies on scarce and costly human resources.

Every second counts. The fastest the operator detects the incident; the best the damage is minimized. Incident detection counts on the capabilities of a human operator, to observe, to analyse (detect and identify) moving objects, and to understand their actions and interactions within the field-of-view (FOV) of the cameras.

In the management of surveillance control rooms, the cameras per monitor ratio or the number of CCTV screens per operator is an important factor. Most of the police forces, in surveillance field, suffer from human resources deficiency. Especially when having a big number of cameras. Consequently, a limited number of operators are responsible to constantly monitor a large area, by observing a single monitor showing multiple streams simultaneously or sequentially, and this is the case of most of CCTV control rooms. However vigilant the operators are, monitoring process suffers from the huge amount of information, which leads to inattention due to fatigue, interruptions and distractions, and physical limits. Police operators cannot keep continuous surveillance effectively. Unfortunately, in such manual system, many incidents are miss-detected. As a result, surveillance videos are often used in passive monitoring or as evidence for post-incidents investigations. These missdetections of important events can be dangerous in critical surveillance tasks such as public places, sensitive locations, airport, and border control surveillance.

Beside this, accessing video data storage is very limited and far away from efficiency when the analysts are working on post-incident investigation. Those video analysts need specific location, specific time and specific incident type and description. Most of the time, at least one of those three is not available or, let 'us say, not accurate. For this reason, it may take a very long time for a human to detect it. Then tracking the involved objects (persons and vehicles) and analysing them is another part of the problem. The analyst should fetch all surrounding cameras to trace each one of those objects, and hopefully uncover all the necessary information about them. The information may be object identification, person description, vehicle description and plate number, etc. In some cases, it may take months to analyse one incident.

To overcome these limitations of traditional surveillance methods, the computer vision and artificial intelligence communities are seeking to develop automated systems for the real-time monitoring and archives investigation of contents understanding like vehicles, people, other objects, actions and interactions.

In a surveillance control room, especially when observing a dynamic scene like public places, motion is the daily basis. As mentioned, motion information stands out as the most important cue to identify the dynamic content of videos. Extraction and analysis of motion information in videos are crucial in automated surveillance video systems. And as daily motion is the regular thing, the most important part of observation or analysis is to focus on non-linear motion like the one we may observe during an interaction between objects in the scene, mainly humans, vehicles, or any kind of moving objects, either deformable or nondeformable. The scene type and environment can be very divers. Moreover, many objects can exist in the scene, and many types of interaction could occur, as mentioned in the ontology.

For most human, describing what is happening in a video is an easy task. For computers, extracting, analysing, and understanding what is happening from video pixels and generating a description is still a very complex problem. A relatively wide panel of works on many fields concerning video description in general and video surveillance in particularly has been published (see the state of the art in chapter I). To simplify the problem, some researches added more assumptions to significantly improve the results but limiting so their applicability in the real world. Most of the researches have specific limitations. They are designed for particular sets of objects, and actions in a specific context. They often lack of a generic multimodal framework to achieve system robustness in multiple contexts, object types and actions performed.

The state of the art for videos surveillance description was thoroughly discussed in chapter I, where we showed that the behaviour understanding and sentences generation approach is the most suitable for our video surveillance description system. Indeed, this approach takes into consideration the need for extracting important behaviour understanding features, and the need for generic expressive structural description. For that reason, in the next section, we share an overview of some of the researches on automated video surveillance, where the interest is to focus on the semantic content features which can be useful for video description. Those features are mainly involved in the behaviour understanding and automated visual surveillance fields.

Then, we present our proposed approach, which is a generic Video Surveillance Scene Description (VSSD), with main focus on interactions between objects, designed to meet the needs of dynamically changing conditions like objects, interaction and context.

IV.2. State of the art

Working on video content analysis and understanding, it is not a field for video surveillance application only, but it trespasses that for many other applications and domains. There is a big need in variety of applications and domains not restricted to surveillance applications, we mention: video indexing (commonly based on text or other) for content-based video annotation / retrieval, human-computer interfaces, computer games, animation and special effects, video editing, analysis of sport athletics, healthcare systems, interactive application and environment (automated houses and cars…), video segmentation, analysis of human conditions (e.g., athletic performance…), etc.

A lot of studies mainly focused on surveillance applications, like person identification, person or car tracking, crowd flux analysis and statistics or congestion analysis [START_REF] Feris | Boosting object detection performance in crowded surveillance videos[END_REF], anomaly detection and alarming [START_REF] Neves | Biometric recognition in surveillance scenarios: a survey[END_REF], access control, interactive surveillance using multiple cameras (tracking objects…), people counting [START_REF] Hou | People Counting and Human Detection in a Challenging Situation[END_REF], behaviour analysis [START_REF] Pantic | Human Computing and Machine Understanding of Human Behavior: A Survey[END_REF] (T. [START_REF] Ko | A survey on behavior analysis in video surveillance for homeland security applications[END_REF] and action recognition [START_REF] Neves | Biometric recognition in surveillance scenarios: a survey[END_REF].

Behaviour analysis and semantic content extraction contains analysis and recognition of motion, actions and interactions between objects. It is, for visual surveillance, one of the most advanced and complex research in image processing, computer vision, and artificial intelligence. The studies in this area focus on the advancement of visual analysis techniques in order to extract the semantic information about regularity or abnormality of the scenes objects behaviours (e.g., human & vehicle).

An automated visual surveillance system which can understand and learn behaviour from observed activities in a video sequence requires a reliable integration of image processing techniques and artificial intelligence techniques [START_REF] Jan | Neural network based threat assessment for automated visual surveillance[END_REF]) (T. [START_REF] Xiang | Video Behavior Profiling for Anomaly Detection[END_REF].

Our main goal is to obtain a meaningful semantic content which can be used for description of what is happening in a monitored area, either in live mode to take appropriate action based on that interpretation, or in offline mode like storing the video sequence with the textual description to provide an easy intelligent access. The description may vary according to the need, context, objects, and intended actions.

For extracting useful content features, many means were used depending on the output specific goal, we mention: object detection and segmentation, object tracking, trajectory analysis, action analysis, activity classification and recognition, and others.

Huge amount on "Behaviour analysis and understanding" studies and surveys are not restricted to only video surveillance systems; Many excellent surveys like (W. Hu, Tan, [START_REF] Hu | A Survey on Visual Surveillance of Object Motion and Behaviors[END_REF] (Vishwakarma & Agrawal, 2013).

(Taha, H. Zayed, E. Khalifa, & M. El-Horbaty, 2014), (Teddy Ko, 2011), (T. [START_REF] Ko | A survey on behavior analysis in video surveillance for homeland security applications[END_REF], [START_REF] Wang | Recent developments in human motion analysis[END_REF], and [START_REF] Kumar | Study of robust and intelligent surveillance in visible and multi-modal framework[END_REF] discuss the general framework and the general architecture of a video understanding system exploiting behaviour analysis.

More recently, since the evolution of many neural network techniques, many of these techniques and algorithms were used in many content extraction fields. Some content extractions were largely improved and achieved satisfied results, other still not mature. As can be clearly predicted, the introduction of these machine learning techniques for this field of research is very promising.

Next, we present the state of the art of some related subjects, concerning object(s) tracking, trajectory analysis, action analysis and recognition, and textual description templates. Also the reader can refer to the Appendix VIII.5 for more related works concerning object detection, object segmentation, object classification and video action analysis. Finally, before presenting our proposed approach, we highlight some of the complexities that face most works when dealing with video surveillance.

IV.2.1. Object tracking and Multi-object tracking

IV.2.1.A. Object tracking

The task is to track moving objects through frame sequences. Object tracking is the process of locating, over time, a moving object. This can be difficult in some cases; depending on the angle, distance and the object speed. Most studies use matching techniques to make sure that the same blob is being tracked in each subsequent frame. Different techniques can be mainly divided into seven main categories, according to [START_REF] Morris | A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance[END_REF] In another approach, [START_REF] Neves | Biometric recognition in surveillance scenarios: a survey[END_REF] distinguish in general, tracking approaches regarding to the tracking technique adopted (Bayesian, Kernel Filter Model / Shape, by-Detection) and the type of information (motion, appearance, and shape) used to model target objects, usually denoted as target representation.

To track moving objects, deep neural networks, especially convolutional neural networks (CNN) were recently proposed. Some promising results are shown in (H. [START_REF] Li | DeepTrack: Learning Discriminative Feature Representations Online for Robust Visual Tracking[END_REF] and [START_REF] Nam | Learning Multi-Domain Convolutional Neural Networks for Visual Tracking[END_REF]. [START_REF] Wang | Visual Tracking With Fully Convolutional Networks[END_REF], proposed to create an object tracker by online selecting the most significant hierarchical features from an ImageNet pre-trained CNN.

Zhai et al. [START_REF] Zhai | Deep Learning of Appearance Models for Online Object Tracking[END_REF]) used a Bayesian classifier as a loss layer in CNN tracker.

As for Nam et al. [START_REF] Nam | Learning Multi-Domain Convolutional Neural Networks for Visual Tracking[END_REF], they trained a multi-domain CNN, for tracking objects, using learning generic representations.

Comprehensive surveys for conventional object tracking can be found in [START_REF] Morris | A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance[END_REF], [START_REF] Ko | A Survey on Behaviour Analysis in Video Surveillance Applications. Video Surveillance[END_REF], and (W. [START_REF] Hu | A Survey on Visual Surveillance of Object Motion and Behaviors[END_REF].

A comparison of methods based on deep learning, but mainly focused on visual tracking, has been presented in (P. [START_REF] Li | Deep visual tracking: Review and experimental comparison[END_REF]. According to Li and al., their comparison shows that, using deep convolutional neural network for tracking, could improve significantly the performance.

IV.2.1.B. Multi-object tracking

To solve the problem of multi-object tracking, one can plan to use the object tracking algorithms, in multiple instances; however this approach requires an additional data association module, as, for example, in the Multiple Hypothesis Tracking [START_REF] Reid | An algorithm for tracking multiple targets[END_REF], or the Joint Probabilistic Data Association Filter [START_REF] Fortmann | Sonar tracking of multiple targets using joint probabilistic data association[END_REF], or the appearance similarity [START_REF] Breitenstein | Robust tracking-by-detection using a detector confidence particle filter[END_REF], or the prediction-based tracking (Particle filter, Kalman filter). Our selected approach ("Motion-Based Multiple Object Tracking -MATLAB & Simulink," n.d.), after detecting moving objects in each frame, uses kalman filter to predict the track's location, for associating the detections corresponding to the same object over time.

Other techniques were proposed, like ObjectTracker, Deform PF-MT, PWP3D, Globally-Optimal Greedy Algorithms, Continuous Energy Minimization for Multi-Target Tracking, Two-Granularity Tracking, GMCP-Tracker, Urban Tracker, BPF, Tracking Interacting Objects, Learning to Track, and many others.

A comparative work of many of the above algorithms is presented in the experimental section IV.4. However, it is important to mention that this comparison was made at an early stage of this thesis. Therefore, some recent studies, based on deep learning, were not included in this comparison, but will be considered in our future work.

Recently in (Ankush [START_REF] Agarwal | Real-Time* Multiple Object Tracking (MOT) for Autonomous Navigation[END_REF], the authors propose a multiple object tracking by using a region based convolutional neural network (RCNN) for object detection and by creating a regression network for generic object tracking.

In [START_REF] Milan | Online Multi-Target Tracking Using Recurrent Neural Networks[END_REF] the authors used a recurrent neural network and LSTM, to perform target state prediction, state updates, and data association.

Another work using deep neural network for Multi-object tracking can be found in [START_REF] Gaidon | VirtualWorlds as Proxy for Multi-object Tracking Analysis[END_REF].

IV.2.2. Trajectory analysis

The trajectory of motion is important for the analysis of a video, and can be widely applied to many domains, such as indexing and extracting a video (W. [START_REF] Hu | Semantic-Based Surveillance Video Retrieval[END_REF], (W. [START_REF] Hu | A Survey on Visual Surveillance of Object Motion and Behaviors[END_REF], video scene segmentation, video semantic analysis. The analysis of the trajectories can help the recognition of the events, actions or interactions between objects.

It is an intermediate level between the low and the high level of analysis. However, the direct modelling of spatiotemporal variations of trajectories is complex because of their non-linearity.

Movement trajectories provide rich information about the spatio-temporal activity of an object. Each trajectory records not only the coordinates of points (the position sequence of the tracked target) and the local directions (direction of the object at each position) on the image trajectories [START_REF] Bashir | Real-Time Motion Trajectory-Based Indexing and Retrieval of Video Sequences[END_REF], [START_REF] Buzan | Extraction and clustering of motion trajectories in video[END_REF], [START_REF] Chan | Detecting rare events in video using semantic primitives with HMM[END_REF]), but also speed and acceleration [START_REF] Hongeng | Video-based event recognition: activity representation and probabilistic recognition methods[END_REF], [START_REF] Wang | Learning Semantic Scene Models by Trajectory Analysis[END_REF]).

An enormous work on the understanding of behaviours, events and actions has been conducted on the basis of trajectory analysis. The majority of these efforts in the field of visual surveillance are focused on similarity and clustering of trajectories [START_REF] Anjum | Multifeature Object Trajectory Clustering for Video Analysis[END_REF], [START_REF] Kataoka | Big Trajectory Data Analysis for Clustering and Anomaly Detection[END_REF], [START_REF] Wang | Learning Semantic Scene Models by Trajectory Analysis[END_REF]; detection of abnormal trajectories [START_REF] Kataoka | Big Trajectory Data Analysis for Clustering and Anomaly Detection[END_REF], [START_REF] Makris | Path detection in video surveillance[END_REF], (D. [START_REF] Makris | Learning semantic scene models from observing activity in visual surveillance[END_REF] ; detection and classification of events (Z. [START_REF] Zhang | Trajectory Series Analysis based Event Rule Induction for Visual Surveillance[END_REF], [START_REF] Piciarelli | Trajectory-Based Anomalous Event Detection[END_REF], [START_REF] Hervieu | Video event classification and detection using 2D trajectories[END_REF]; and scene modelling (Points of Interest (POI) where interesting events happen (entry / exit, stop), activity paths (PA), junctions, roads) (Xiaogang [START_REF] Wang | Learning Semantic Scene Models by Trajectory Analysis[END_REF], (D. [START_REF] Makris | Learning semantic scene models from observing activity in visual surveillance[END_REF], [START_REF] Black | A hierarchical database for visual surveillance applications[END_REF], and (Sangho Park & Trivedi, 2007), where, later discussed in IV.3.3, a similar outputs were presented with our approach.

A recent work [START_REF] Dogra | Smart video summarization using mealy machine-based trajectory modelling for surveillance applications[END_REF] proposed a method using a finite state machine to analyse the trajectory and the instantaneous velocity to detect what they named it "event(s)-of-interest", means when an interesting variations occur. These events of interest used to help in summarizing the scenes.

In our approach we used a similar concept to detect important variation, but not only in velocity and trajectory, but also directions, surface, Hu moments, and deformability, to trigger the description at such moments.

IV.2.3. Action and Activity classification and recognition

Another important area of research today is Action and activity classification and recognition. Its goal is to automatically analyse ongoing activities from an unknown video. It includes the analysis and the recognition of patterns to infer higher level description of objects actions and interactions. Also, it is the process of recognizing the actions to know and understand what is happening in a given context [START_REF] Loy | Activity understanding and unusual event detection in surveillance videos[END_REF].

Taking advantage from these four categories, we present in the chapter II a deeper perspective for conceptually categorizing the type of video surveillance scenes into 15 categories.

Human activity recognition approaches is mainly divided into two categories. (1) The traditional representation-based approach based on the feature detectors and descriptors, e.g. trajectory, and Scale-Invariant Feature Transform (SIFT), then for action recognition, a generic trainable classifier is applied; (2) Learning-based representation approach, which is a recently developed approach with capability of learning features automatically and directly from the images and frames. In this approach no need for feature detectors and descriptors.

A-Traditional human activity recognition approaches: or approaches based on handcrafted local features, was classified by (Aggarwal & Ryoo, 2011) into two main categories: Non-hierarchical and hierarchical approach. 1-The non-hierarchical approach or single layer approach recognizes the human activity, based directly on image sequences, by matching the activity with already known ones. This approach mainly used for simple and short activities such as periodic activities and primitive action (jumping, running, waving, etc.). It is divided into two sub-classes: sequential approach and space-time approach. 2-The hierarchical approach is usually used for complex human activities such as multiobject activities, human-object interactions and group activities. It represents these by describing them in terms of simpler activities. It can be classified into three categories: syntactic approaches, statistical approaches, and description-based approach.

B-Learning-Based Action Representation Approach

On other hand, this recent approach is based on the last progress on learning field. It has capability to learn the feature automatically from the raw data (frames Interesting works is being handled newly in this field [START_REF] Pham | Learning to Recognize 3D Human Action from A New Skeleton-based Representation Using Deep Convolutional Neural Networks[END_REF]. Also, important survey can be seen in [START_REF] Asadi-Aghbolaghi | A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences[END_REF]. But so far, as mentioned before, a lot of work with the learning models has been done on images and classification in images, where those algorithms have achieved very good results. In videos, some work dealt with gestures, actions and group activities, and promising results were found. However, fully data-driven deep models, referred to as "black-box", have also some limitations in videos [START_REF] Sargano | A Comprehensive Review on Handcrafted and Learning-Based Action Representation Approaches for Human Activity Recognition[END_REF], where the performance of the learning-based methods solely is still not up to the mark. This is mainly due to the unavailability of huge datasets for action recognition unlike in the object recognition where huge dataset exists.

For action and activity classification and recognition, till now, either in both approaches (traditional and learning), not very distinctive results have been done on the level of interaction. Some studies suggest that unsupervised learning it is going to be far more important in the long run. Since the human and animal learning is mostly unsupervised.

In our work we focus mainly on the interactions. Our intention is to detect the existence of interaction between two objects, and to classify it according to its types. As using learningbased approaches for videos at the stage of features extraction is still not satisfying and does not take into consideration the temporal nature of interactions, we choose to extract meaningful feature appropriate for our classification. Nerveless, we used for our classifications the learning-based approaches in both faces, Non-Deep and Deep learning, see section experiments IV.4.5.

IV.2.4. Textual description templates

In the state of the art of the behaviour understanding and sentences generation approach, used for our video surveillance description system, shows that many sentence generations are based on handcrafted structured templates. These templates differ from work to another according to needs. Some of these templates are presented in Table IV-1: Many concepts and terms (highlighted in bold) in these templates are found interesting for our system, we mention the location, frame range, variety of speed, and the direction. Nerveless, our structured templates is more developed, see sub-section IV.3.8.

IV.2.5. Complexity of the video

A lot of success has driven the image processing field to converge toward the learning models. While in videos field in general, and specifically in video surveillance field, still did not achieved what is needed and expected. As working on videos is a very complex problem due to the diversity of scenes, we mention:

-The context place type and scene location: the place where the surveillance can occur.

This place can handle two main types: Indoor and outdoor. Indoor places are like Airports, Banks, schools, governmental buildings, metro stations etc. Outdoor places are like City streets, Circulation, Gardens, Parking, Squares, etc.). -The Environment: as it is subject to weather conditions, reflections, and irregular lighting changes according to daylight. -The number of objects: as the scene can handle many objects (crowd or groups), two objects, one object with partial or complete occlusion or deformation. -The object types: as the object can be a human, animal, plant, machine, inert object, and in extension any element composing an urban or a rural landscape. -The variety of actions and interactions: it is an open field of gestures, actions and interactions, in each activity, viewed from different view angles, at different scales.

Significant research and advancement in solving these difficulties have been achieved. As seen in the literature, some research, to simplify the problems, added more assumptions. This may have improved the results significantly but it limited and restricted its applicability in real world. The algorithms developed therefore were designed for a particular type of objects, a specific context, and some particular actions.

IV.3. Proposed Approach

An efficient way to encounter the difficulties of varieties of scenes, object types and actions performed, may be by proposing a generic non-contextual approach. This chapter proposes a new approach for a generic context-independent textual description of the scenes in real-world surveillance video. We propose new representations for the sentences based on well-structured templates, which can be applied to generate incidents scenes description similar to ones used in police investigators reports. Our approach is based on the ontology described in chapter II, which combines low-level primitives of objects basic features, to allow deriving more meaningful high-level information; and this made it more methodologic, and easily expendable. This general approach is not restricted to a context, object or interaction type; in the contrary, it can be applied to any of the scene types at the level of application. In plus, we introduce additional new concepts, concerning mediation and action at a distance, and a new manner of segmenting video and categorizing the scenes, in sub-sections IV.3.4 and IV.3.6. This approach is named Video Surveillance Scene Description approach or VSSD.

The key idea behind our approach is to leverage meaningful content features from the scene for better understanding and an appropriate scene description. These features are well selected, for the real cases and real need of police operators and investigators. They are considered useful in generating alerts, enquiring the video surveillance footage, and inferring textual sentences.

Our main focus was on the characteristics and behaviours of objects, and the interactions between them, mainly when having two objects in the scene. To accomplish that, many phases, listed below, were made:

 First, we detect, segment and track the moving objects in the scene.  Second, we segment the video based on the number of objects moving in the scenes.  Third, objects are classified as deformable and non-deformable. 5. Classify the scene.

6. Extract the different features.

7. Classify interaction/not, then physical/far, and peaceful/aggressive. 8. Identify key moments to fill activity matrix.

9. Generate the textual description sentences which follow our templates.

 Fourth, we generate an activity map, highlighting the routes and hot spots in the FOV, and indicate background changes.  Fifth, we classify the scene.  Sixth, we extract the different features.  Seventh, we detect the existence of interaction or not by classification, then we classify whether the interaction is physical or far, and finally whether the interaction is peaceful or aggressive.  Eighth, we identify the key relevant moments, to fill an activity matrix.  Finally, we generate the textual description sentences which follow our templates.

However, each of the phases listed above, can have as input one or many phases' results; also, its output can feed one or many phases. The proposed approach diagram that can show these relations is shown in Figure IV-1.

The presented description of the video surveillance scenes VSSD, take into consideration the main needed information when an incident occurs. In real cases, when an incident occurs, five main points are mainly needed for a scene description, also known as the five Ws (Who, what, where, when and why (is replaced by how)):

1-WHO Who is the concerning object, which the description should focus on (especially, when there are many objects in the scene)

2-WHAT

What the objects are doing in the scene: movements, actions, activities, etc.

3-WHERE

In which location, in the case of video, in which position in the frame.

4-WHEN

At what trench of time the incident occurred

5-HOW

How did the objects (the who) perform their doings (the what). The circumstances and the way the person perform the crime is one of the most important clues for the decree (judgment), for example: "A car speeds up toward a person and hits the person", is very different than: "A car slow down, change direction away from the person and hit the person", because the second may indicate the intention of the car driver to avoid the person.

For adaptable description output, we left the verbosity frequency at many levels of scene description to be controlled, according to the preference, by the user. The presented approach kept its modularity of analysis tools, allowing improving the produced description at any time. The semantic content can be easily increased by adding information about the context. Next, we present a detailed explanation of each of the stages.

IV.3.1. Segmentation and tracking algorithm

The objective of video object tracking is to follow a targeted object through the sequence of video frames. The segmentation, in turn, classifies the pixels of a video image, separating the foreground from the background. Tracking and segmentation are very important operations, since all the following operations (features extraction and interaction classification) rely on them. Bugs and bad decisions, in these two operations will affect the quality of the final results.

Many efforts have been done in those two fields by the research community, and a lot of problems remain, especially in the case of occlusion. Most of the segmentation and tracking algorithm will be confused between the two objects in this case.

Tracking and segmentation algorithms are not the main core of this thesis. They are considered as being only a tool. As many works have been done in this field, one can assume that we have a tracking and segmentation algorithm which provides a good measurement for further processing in the approach. But in our approach, we wished to deal with realistic results in order to measure their impact on video surveillance automation systems. For that purpose, we searched available tracking and segmentation algorithms, seeking for a one that can provide us with acceptable results for further interpretation. We mainly focused on the available algorithms, able to perform multi-target/object tracking and segmentation of any object types, as we need to track and segment two objects in the scene at the same time. After a comparison of more than 20 algorithms, we found few algorithms able to deliver all of the mentioned conditions. For more detail, the reader is referred to the experimental section IV.4. We chose the algorithm for segmentation and multi-object tracking provided by Matlab, called "Motion-Based Multiple Object Tracking". This algorithm is based on two main steps, according to ("Multiple Object Tracking Matlab," n.d.): 1-Detecting moving objects in each frame: by using a background subtraction algorithm based on Gaussian Mixture Models. For noise filtering, morphological operations are applied to the resulting foreground mask. Then, using blob analysis the algorithm detects groups of connected pixels, which are probably corresponding to moving objects. 2-Tracking the moving objects from frame to frame: by, first, assigning detections to tracks using "Kalman filter" for motion estimation and prediction. Then, initializing new tracks based on unassigned detections, confirming and updating existing assigned tracks, coasting existing unassigned tracks, and finally deleting unassigned tracks, for a too long time.

We add to the algorithm the ability to draw the trajectories of the objects centroids. Samples of results of applying the algorithm on the Fight_RunAway2 scene from CAVIAR Video Sequence ("CAVIAR," 2004) are shown in Figure IV-2.

It is important to mention that the selected "Motion-Based Multiple Object Tracking" algorithm may not be the best one, but it is the simplest, most available, suitable and easy to use one. We can easily replace it whenever we find a better one. We chose this "plug and play" principle for the sake of flexibility.

IV.3.2. Object classification

In our ontology, we differentiated between two general sub-classes of an object: deformable and non-deformable objects. When an observer in a surveillance control room is focusing on an area of interest, the first thing to distinguish, if an object appears in the frame, is its deformability. As mentioned, from surveillance point of view, non-deformable objects actions during an interaction are easy to detect, analyse, understand and maybe predict. When deformable parts of an object move freely in unpredicted way, the analysis becomes more difficult, even for a human brain.

Our main goal is to have an abstract description, which can be applied in any scene type such as circulation, city surveillance, elder house, hospital, borders, zoo, etc. The object can be a human, an animal, a machine or even a plant. We could not find, for our knowledge, a generic algorithm that can segment any object type in any scene, under any circumstances (occlusion, lighting, perspective…), in its semantic sub-components with a very high accuracy. Therefore, we did not try to go deeper into analysis of sub-object segments in the presence of interacting deformable objects.

Recently, for classifying the objects extracted from the scenes after segmentation, many existing algorithms with deep neural network (YOLO v3, Mask R-CNN) can perform good results, especially when it is restricted to a particular scene type having a known number of objects types. As our description is meant to stay on abstract general level, we did not consider applying any of those algorithms. But, in a later stage at the level of application, adding a classification tool to our algorithm it would be very favourable, especially when it is contextual-based where the scene and objects types are well known, which make it simple to learn such algorithms and to have very accurate results. A worthy push for such algorithm, if it has as an input for classification the object is deformable or not.

Consequently, as a first step of analysis and classification of object interactions, we can easily check whether the blobs encasing each one of the objects, resulting from the tracking algorithm, refer to deformable or non-deformable object, following the method mentioned in chapter III. Nerveless, for the reason of having good assessment for the interaction classifications approach and not cumulating, with the mentioned approach, the errors produced by the object classification algorithm, we added the manual classification of the objects, at the level of experimentation, as an input for the system.

IV.3.3. Background change detection, and activity hot spots localisation

After segmentation and tracking, we propose a new simple and efficient way to highlight first the background change detection, and second the routes and activity hot spots localisation by generating an activity map and background model.

As we want to maintain the modularity of our approach, we chose to not use the background model generated by these pre-processing tools, so we build our own model. To have a model for background can help to detect all potential changes made by objects in the background, for the goal to differentiate between the many scene types, as mentioned in sub-section IV.3.6.

To estimate the background model, our method, uncommonly, is non-supervised method. Apart from that, the main target from highlighting the routes and activities spots is to help understanding the scene by indicating the most used areas in the Field of View (FOV), such as the main routes used by the moving objects or the spots for activities like their interactions. Following that the capability of indicating important clues in the trajectories of the moving objects when shifting from an area to another, triggering by that a description moment (see sub-section IV.3.8).

To this end we present a new activity map 𝑀 𝑎 . It is a matrix representing the temporary cumulative occurrence (till the current frame), where each value represents the number of times this pixel has been taken into consideration through the analysed sequence. This activity map is initialized according to that definition: 𝑀 𝑎 (𝑥, 𝑦, 0) = { 0 𝐼𝑓 𝐼(𝑥, 𝑦, 0) 𝑏𝑒𝑙𝑜𝑛𝑔𝑠 𝑡𝑜 𝑎𝑛 𝑜𝑏𝑗𝑒𝑐𝑡 𝑖𝑛 𝑓𝑟𝑎𝑚𝑒 0 1 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

And the matrix of temporal activity is defined as:

𝑀 𝑡𝑎 (𝑥, 𝑦, 𝑡) = { 0 𝐼𝑓 𝐼(𝑥, 𝑦, 𝑡) 𝑏𝑒𝑙𝑜𝑛𝑔𝑠 𝑡𝑜 𝑎𝑛 𝑜𝑏𝑗𝑒𝑐𝑡 𝑖𝑛 𝑓𝑟𝑎𝑚𝑒 𝑡 1 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

Then for each frame of the sequence, we update the value of the activity map as follow:

𝑀 𝑎 (𝑥, 𝑦, 𝑡) = 𝑀 𝑎 (𝑥, 𝑦, 𝑡 -1) + 𝑀 𝑡𝑎 (𝑥, 𝑦, 𝑡).

Based on this activity map, we calculate the background model consequently we detect the background change and we calculate the activity map following the routes and activity hot spots localisation.

IV.3.3.A. Changed Background detection

Even for one camera, many background models may be needed, especially when having long time of surveillance videos. This may be due to light changes, especially in outdoor scenes sunlight changes, or background inert objects displacement when adding or taking objects for long time from the background.

To detect all the changes applied on the background, we calculate a temporary background model based on the activity map presented in (eq. IV-4).

This temporary background model is a cumulative background (till the current frame at the moment), where all pixels in the past frames are taken into consideration except the one belonging to objects. The concept is to compare the current frame (all pixels not (eq. IV-1) (eq. IV-2) (eq. belonging to an object) with the temporary background to detect if there is a big change occurring.

This temporary background matrix 𝑀 𝑡𝑏 𝑖s initialised that way: 𝑀 𝑡𝑏 (𝑥, 𝑦, 0) = 𝐼(𝑥, 𝑦, 0) . 𝑀 𝑎 (𝑥, 𝑦, 0)

Where "." is a scalar multiplication.

Then for each frame of the sequence, we update the value of the matrix as follow:

𝑀 𝑡𝑏 (𝑥, 𝑦, 𝑡) = (𝑀 𝑡𝑏 (𝑥, 𝑦, 𝑡 -1). 𝑀 𝑎 (𝑥, 𝑦, 𝑡 -1) + 𝐼(𝑥, 𝑦, 𝑡). 𝑀 𝑡𝑎 (𝑥, 𝑦, 𝑡))/ 𝑀 𝑎 (𝑥, 𝑦, 𝑡)

NB: 𝑀 𝑎 (𝑥, 𝑦, 𝑡) may contain zeros, and so, all the corresponding values in 𝑀 𝑡𝑏 (𝑥, 𝑦, 𝑡) were not calculated but replaced by zeros.

Finally, once the whole sequence of N frames has been processed, the background model is generated as being the average image without moving objects, and equal to 𝑀 𝑡𝑏 (𝑥, 𝑦, 𝑁) . An example of the background model can be seen in Figure IV-3.

To determine when a background has changed and the moment when a new background model is required, we compare the temporary background model, each time, with the current frame to determine if a percentage 𝑝 𝑝 of pixels changed significantly their values. The algorithm fills out a "background changes" vector, referring to those frames.

𝑉 𝑏𝑐 (𝑡) = {

1 𝑖𝑓 𝑝 𝑝 𝑜𝑓 𝐼(𝑥, 𝑦, 𝑡). 𝑀 𝑡𝑎 (𝑥, 𝑦, 𝑡) 𝑎𝑟𝑒 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑡 𝑓𝑟𝑜𝑚 𝑀 𝑡𝑏 (𝑥, 𝑦, 𝑡). 𝑀 𝑡𝑎 (𝑥, 𝑦, 𝑡)) 0 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

Example of a background changes vector:

( [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [1] [0] [0] [1] [0] [0] [0] [0] [0] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] ….).
All the backgrounds detected as changed ones are not taken into consideration when calculating the temporary background model. As miss classified background change can occur, we use again the temporal-consistency algorithm proposed by [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF] to correct this artefact. And the example above after filtering will be:

( [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] ….).
The change in the background can be temporary or permanent relatively. Many reasons and actions could be behind that. To indicate which case it is, a threshold (changed sequence 𝑐𝑠 𝑏 ) should be set designating for how much time (number of frames) the backgrounds remain detected as changed in the filtered vector before considering the change as permanent.

If the background change is temporary, this may imply a further investigation to indicate the reason and the location of the change. The detection of temporary change may help to classify the scene, as done later in the sub-section IV.3.6.

If a permanent change were detected, this indicates a new sequence of frames began. Subsequently a new background model is required, and all the above calculation of the activity map 𝑀 𝑎 and 𝑀 𝑡𝑎 and the background temporary matrix 𝑀 𝑡𝑏 and 𝑉 𝑏𝑐 should be reset.

NB: Determining the above-mentioned thresholds 𝑝 𝑝 and the 𝑐𝑠 𝑏 is very verbose and depends on too many factors; we mention the scene type, the FOV, and the user need. For (eq. IV-4) (eq. IV-5) (eq. IV -6) example, if the user needs to detect small object left or taken, like a bag, in an indoor airport hall scene, this 𝑝 𝑝 should be set to very small value, later 𝑐𝑠 𝑏 can be set to several seconds only. On the contrary, in a traffic scene, detecting if a car is parked on the road, this may be satisfied with bigger 𝑝 𝑝 threshold, later 𝑐𝑠 𝑏 can be set to several minutes.

IV.3.3.B. Routes and activity hot spots localisation

The routes and activity hot spots may extremely differ between one location and another and one sequence to another, especially when having non-constrained interactions in the real videos.

To localise and present the routes and activity hot spots on one map, the activity map, for the whole sequence of N frames, a simple calculation can be done. Once the whole sequence has been processed, we generate the activity map 𝑀 𝑎𝑚 as follow:

𝑀 𝑎𝑚 = 𝑀 𝑎 (𝑥, 𝑦, 𝑁)/𝑁
For each of the scene frames where its background was detected as changed are not taken into consideration when calculating the activity map 𝑀 𝑎𝑚 . A permanent change detection of the background implies a new calculation of the activity map 𝑀 𝑎𝑚 .

In Figure IV-3, where in the activity maps we can see several degrees of grey. The darkest grey represents the smaller value, which means more objects pass by these points. In the activity map, the low values represent flows of moving objects in the FOV. To find routes and regions of activities, we use a simple method in three steps:

1-Pixels quantification: on the activity map, we did a linear quantification of pixel intensities into four levels, see Figure IV-6 shows the routes in green and the activity hot spots in blue and red, of scene "Fight_RunAway2", taken from the database "Caviar" ("CAVIAR," 2004). We notice in The result of the above steps populates a scene model which can be used in the description phase, sub-section IV.3.8 or used for detecting anomalies.

IV.3.4. Video Segmentation

After segmenting and tracking video objects, we perform simple automatic segmentation of the video into sub-scenes. These sub-scenes take into account the number of objects. Four types of sub-scene are taken into consideration, see in Figure IV-7: 1-No objects. 2-One object. 3-Two objects. 4-Many objects.

Figure IV-7: Simple video segmentation according to the number of objects in the scene

The actions can be performed by one object (self-action or interactions with the background), two objects (interacting with each other) or many objects (many interacting objects or crowd activity), and in each case many types exist.

Having one person in the scene performing gestures, or interactions with the background, is the subject of many researches, and some had already good results. This is also the case when many objects in the scene are performing group activity. Many interesting works exist in the field of crowd detection and analysis. The crowd analysis is an interesting field; it can help in managing the flow. However, this field focuses mainly on the crowd as one entity to study the flow, the number of persons, and others; and do not focus on incidents and interactions. In this thesis, since our high interest is to detect and analyse incidents in public places, we will focus mainly on sub-scenes corresponding to two interacting objects.

IV.3.5. Feature extraction

Selecting and extracting the right features has an important role in improving both the efficiency and accuracy, and can guide the algorithm to powerful results. Automatic extraction of features directly from images and videos to feed machine learning algorithms, as an end-to-end solution exist. As known so far, the learning models and algorithms are recent efforts and have achieved promising results on gestures, actions and group activities recognition. However, few researches have been done on objects interactions. The obvious reason is the complexity of the task even for a human brain. It is challenging to analyse and understand an interaction in some cases; uncountable types of interactions can happen, and each can occur in many different ways. Even the reaction of an object in the interaction varies from case to case and may be unpredictable.

An improvement in the domain of object interaction analysis and understanding can be done, but it is still a challenging task, and a recruit to traditional feature extraction may be the best solution. In plus, the learning-based approach for videos is still not satisfying and does not take into consideration the temporal nature of interactions.

In our approach, we join the meaningful features extracted from objects, listed in this section, with the power of deep learning as it will be shown in experiments described later. We tried to extract the realistic features observed by human being when two objects are interacting. Later we extend the set of input features, using operators that generate combined features. These simple and combined features are intended to capture some higher-level ones about objects displacements and/or interactions to study a potential correlation. They are, for example, the relative distance between two objects, the Hu moments differences, angles of displacements, and the speed of the movement of an object, etc.

In two objects scenes, the duration of the shortest interaction considered were of one second length. Five types of features were extracted, spatial, temporal, inter-objects, inter-frames and trajectory features:

A. Object spatial Features: after the segmentation, in each of the scene frames, the features, shown in -The spatial features of object one and object two -The temporal features of object one and object two -The inter-objects features for object one and object two

For almost each of the features, we extract the derivative and second derivative, if it can be applied. Then for each of the features (f), its derivative (df) and its second derivative (ddf), we find seven global inter-frames features (see Table VIII-16 in the appendix VI which are the minimum, the first, the last, the middle, the average, the median and the STD, normalized by the maximum value).

E. Trajectory Features:

The last set of features is related to trajectory. For that, we compute the trajectory of object one and object two centroids through window frames, and the trajectory of the middle points between object one and object two centroids through window frames. For each of those three trajectories, we apply three smoothing filters: the average filter, the first order prediction filter and the second order prediction filter. For information on these filters the reader may refer to the appendix VIII.6.

Finally, for each of those three original trajectories (object one, object two, and middle trajectories of objects), we generate theses new filtered trajectories. So, applying that, we obtain nine trajectories. At last, for each of those new trajectories we calculate two features, the standard deviation (of the distance between the filtered position and the one corresponding to the centroid), and the largest distance, rendering so some information about the smoothness or the chaotic aspect of the trajectories.

Finally, for each window, we have one set of features containing the inter-frames features and the trajectory features, see Figure IV-8. In all we have for each window 2498 features (divided between 926 inter-frames of spatial features, 1008 inter-frames of temporal features, 546 inter-frames of inter-objects features and 18 trajectory features).

Figure IV-8: Features extracted from a window of N frames. Each set of features at an arrow starting point feeds the set of features at its terminal point.

These extracted features are well chosen to be meaningful so they can be used for many reasons:

-Classifying different interactions types: they are suitable to detect the existence of interaction between the objects, and to classify if the interaction is far or close, aggressive or peaceful. -Generating alerts: they contain very important features to trigger the alerts when needed in real scenes.

-Querying intelligently the archives: they contain very important features and most used in real cases to count on when investigating intelligently the archives. -Generating the textual description: they are suitable to fill the structural templates useful for the real cases.

IV.3.6. Scene type classification

In our ontology, we proposed to classify scenes type into 15 types according to the number of objects and the type of action and interactions performed. In the Table IV-2, we present a simple way to discriminate between 13 of 15 types according to background changes, the number of objects and objects characteristics (features) changes before and after performing the action. Let us consider here that a scene is a temporal window of a predefined length centred on a location where a change occurs (number of objects, background …).

For the number of objects, before and after the action, we use the results of the video segmentation (see sub-section IV.3.4). Concerning the background change, the results are taken from the background change detection, as explained in sub-section IV.3.3.

For features changes of moving objects, we mainly focus on two characteristics, Hu moments (Hu_objs_diff) and surface (S1_objs_diff), mentioned in sub-section IV.3.5. The following table summarizes the expected evolutions of those features in different cases.

For example, the classification of the type "1 Object  1c" corresponds to an object left in the scene.

IV.3.7. Interaction classification

In a surveillance control room, when observing a dynamic scene like public places, motion is the daily basis. And as seen in sub-section IV.3.6, various scene types with objects performing actions can be present. But the most important part of observation or analysis for public place surveillance is to focus on irregular actions like unusual interaction between objects in the scene, mainly humans, vehicles, or any kind of moving objects, either deformable or non-deformable. In fact, two objects interacting can be of high interest when analysing incidents in public places.

And so, after detecting the existence of two objects in the scene, deeper analysis and classification is done in this section in order to decide, mainly, whether there is interaction or no. This situation has been labelled as 2 Objects  2a and 2 Objects  2b in our ontology. There is a lack of scientific studies targeting this special case.

There are many points here to be considered. First, many types of objects exist. Second, the interaction can be distant, physical or both at different consecutive times. We consider it distant, when there is no physical touch between the objects and, on the contrary, as a close one. Third, unlimited types of interactions can occur.

Working in a CCTV control room, especially when dealing with security, every second counts. The sooner the operator detects the incident, the quicker potential damages can be minimized. Most of the interactions start as "distant". For example: saluting before shaking hands, heading to a fight, shouting before hitting, avoiding accident then fighting, etc.

For that, it is important to detect the moment a distant interaction starts. Popping up cameras when distant interaction starts before a fight or accident, can give more time to the operator to focus with PTZ (movable camera) on what is important before losing it, for example, the plate number of a car or the description of the offender before he runs away, etc. In many incidents, the difference between revealing the truth or not by the operator is measured in seconds.

To deal with these problems, especially the three mentioned points, we took the following strategies. The first point concerning objects types, as mentioned, we choose to classify the objects in two main categories, deformable and non-deformable according to chapter III, which is the most visual important feature that can affect the way an object can perform the act. Later, more features are needed for the deformable objects concerning the sub-objects, and further sub-categories can be classified for deformable and nondeformable objects, see chapter II. For the second point, after creating a sliding window and extracting objects and interactions features from that window, many classification algorithms were tested, mentioned in section IV.4, to classify the existence of interaction between the two objects inside the window. We use a neural-based deep model algorithm as the classification algorithm. The first part of the model learns inter-frames and trajectory features. Then, the second step consists in using these learned features to train the algorithm in order to classify the entire window sequence. Later, for the entire scene, after having classified each window as showing an existing interaction or not, then the same concept used in chapter II for temporal-consistency analysis proposed by [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF] can be applied to correct the wrong classified windows.

After classifying the scene where there is an interaction between the two objects, then third point is aiming at going deeper into more layers on analysing this interaction. We add more sub-categories like: aggressive, non-aggressive, bad influence, good influence, etc.

In this study, we choose to classify the interactions, when exists, into objective-likely classification as distant or physical, and more subjective classification as aggressive or nonaggressive. Using the selected features, mentioned in section IV.3.5, the objective is to detect and classify potential interaction between two objects in a temporal window, and later to classify if it is distant or physical interaction, aggressive or peaceful.

To perform these tasks, then, after the dataset scenes selection, see subsection IV.4.1, and feature extraction, see sub-section IV.4.3, the datasets were prepared and pre-processed, see sub-section IV.4.4. After that, many tests on different machine learning algorithms (classical algorithms or Neural networks) were made, see section experiments IV.4.5.

Consequently, we choose different multi-layered Deep Neural Network DNN. For implementation, simple Feedforward networks called Pattern recognition networks in MATLAB and Simulink environment ("Pattern recognition network -MATLAB," n.d.) was learned. In order to achieve the desired outputs, several tests were made after altering this model to handle deeper architecture (by adding more layers) and by determining the best parameters that maximise the results such as the activation function, the training method, etc.

IV.3.7.A. Interaction vs non-interaction classification

Our objective here is to detect if an interaction exists between the two objects in the selected scenes. For each of the windows, the algorithm extracts the features and feed the DNN to attribute the value 1 or 0 to the window according to the existence of an interaction or not between the two objects, see Figure After tests, the chosen algorithm is the seven-layer pattern recognition network, a feedforward network composed of fully connected layers, six hidden layers and one output layer. Each hidden layer contains 586 neurons. The six hidden layers' activation function is the Log-sigmoid transfer function and a softmax transfer function in the output layer. Other parameters are shown in the experiments section IV.4. As result of training this algorithm and testing it with approximate of 14902 records and 2305 features, takin from 285 scenes, the accuracy of the test set achieved 87.5% after 325 epochs.

IV.3.7.B. Distant vs physical interaction classification

Most of the interactions in the real word start from a distance (saluting, fight …), detecting if an interaction is distant (far) or physical (close) have big importance. But working on distant interaction, is not an easy case, and as for our knowledge, we couldn't find any research taken into consideration this case.

For that, if interaction is detected between two objects in the windows, here, we classify if each of those interactions in the selected scene is distant or physical. After tests, on many machine learning algorithms, the chosen algorithm is the four-layer Pattern recognition network of fully connected layers, three hidden layers and one output layer. Each hidden layer contains 426 neurons. The three hidden layers' activation function is the Log-sigmoid transfer function and a softmax transfer function in the output layer. Other parameters are shown in experiments section IV.4. As result of training this algorithm and testing it with approximate of 7079 records and 2303 features, the accuracy of the test set achieved 93.7% after 102 epochs.

IV.3.7.C. Aggressive vs Non-Aggressive interaction classification

Detecting an interaction if it is aggressive or peaceful is very important especially in the domain of public safety and security. Moreover, detecting distant aggressive interaction can alert the CCTV control rooms' observers at early stages, giving them precious time to act. As mentioned, in an incident, an offender may leave the scene in seconds; if the observers were alerted at the right time they may use the movable cameras to catch the plate number before he leaves the crime scene.

For that, if interaction is detected between two objects in the windows, here, we classify if each of those interactions in the selected scene is aggressive or non-aggressive. After similar tests as above, the chosen algorithm is the four-layer Pattern recognition network of fully connected layers, three hidden layers (with Log-sigmoid) and one output layer (using softmax). Each hidden layer contains 546 neurons. Other parameters are shown in experiments section IV.4. As result of training this algorithm and testing it with approximate of 6703 records and 2303 features, the accuracy of the test set achieved 93.8% after 88 epochs. Finally, after detecting for each window the existence of interaction (or not) between two objects in the selected scenes; also, whenever an interaction is detected, if it is distant or physical, and if it is aggressive or peaceful; all the results of each type of classification in the scene is collected and concatenated in 3 sequential classification vectors, see Figure As the output vectors may contain wrong decisions, for that, we applied the method proposed by [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF] to correct these marginal wrong decisions. After this post-processing step, the resulting vector will be more coherent and presents sequences of 0's and 1's indicating (the 1's) the time when the interaction starts and when it finishes.

IV.3.8. Scene analysis and description

"At frame 392: Deformable object 2 moves, in F spot, on the right middle of the inside area of the camera field of view, heading immediately down right, toward the object 1, no big change occurring respectively on its shape, and big changes occurring respectively on its surface having now bigger one, and having respectively slight increasing of its Speed.

"The two objects are respectively receding; a physical peaceful interaction occurs between them."

Example of scene description and of object interaction description.

Having such a description based on activity analysis of a scene can be very beneficial for video surveillance in live mode, for example, the system, with some key text description like "aggressive", can raise alerts, and help in the post processing of this situation for dispatching patrols, etc. Also, specific information of object type, shape, displacement, location, and/or direction, interaction type, can be searchable using textual queries, and this could help investigators to solve many problems and generate textual reports. In Appendix VIII.8, a sample of police report is shown.

When dealing with real incident cases, a big need appears is to have an answer quickly on the five mentioned "W"s (Who, what, where, when, and why), plus the How. Answering those can set the right frame to scene description. For that, in our method we describe the scenes, in an abstract way and no matter what the context is. We mainly focus on scenes having two objects, but the same method can be used for different types of scenes. In each scene, we want to describe textually what is happening in many sentences localised along a temporal dimension.

So, from the extracted features we analyse in the scene: -What is related to each of the two moving objects: on many states, at relevant moments and positions in the frames, like type, shape, and displacement. -What is related to the two moving objects together: on many states, at relevant moments and positions in the frames, like inter-objects distance, position and direction. -What is related to the objects interaction if exists: before, during (distant and physical) and after, and if the interaction is aggressive or not.

Then, differently than the works presented in [START_REF] Dogra | Smart video summarization using mealy machine-based trajectory modelling for surveillance applications[END_REF] where the authors extract key frames only from the trajectory, we extract key moments from those several characteristics, where the object/interaction corresponding states, show irregularity to trigger the description. Triggered by these key moments, a scene activity characteristics matrix of corresponding characteristics is filled.

Finally, we apply logical rules on the scene activity characteristics matrix to generate textual description sentences by filling new proposed structured templates.

IV.3.8.A. Scene key moments

We chose several characteristics representing the behaviour of the objects in different aspect. These characteristics are then represented using a graph-based pattern discovery, from which we extract key moments, to generate the proposed description. For better understanding these key moments, we show some of the key frames, for the scene "LeftBox" from the database ("CAVIAR," 2004) a. The object type: as mentioned in chapter III, a so-called deformable object type can change its behaviour between non-deformable and deformable, while a nondeformable object is not able to do so. These changing moments are considered key moments.

b. Shape related: in the scene, many shape features are relevant to trigger the scene description. For each of the two objects, we analyse the next two features, searching for big variations on their values as key moments: i. Invariant moments or "Hu moments" are shape features used to generate the temporal feature "Hu_objs_diff" from Appendix VIII. ii. Position to an area of interest: important moments to describe the objects states are when they enter or exit areas of interests (routes and activity hot spots) located in the scene frame, seen in section IV. IV3.3 For example, in the whole scene "LeftBox", taken from the database "Caviar" ("CAVIAR," 2004), in Figure we identify different areas according to the regions intensities taken from pixels quantification and morphological filters of the activity map, and projecting the results on the background model, all seen in section IV.3.3 These areas represent the marginal areas (higher intensity region in white colour), regular routes (in green colour), and 2 levels activity hot spots (in blue colour and in red colour for lower intensity region). Then we label with a letter each area from the highest region intensity to lowest one, except the marginal areas. Here we have 3 areas: A, B, and C. 2-Inter_Object characteristics: in the scene, many inter_objects features are relevant to trigger the scene description. We chose only two of them: a. Distance between the objects: Its local minima or maxima can help to decide when generating a description. See Figure IV-18. b. Position and direction compared to the other object: here, key moments are indicated in two cases: first, when an object starts and ends going "toward" the other object; second, when that object starts and ends getting "away from" that other object. Object 1 is considered heading toward the object2, when its vector direction is pointing close to the position of the object 2; otherwise, when this vector is in opposite direction, object 1 is considered getting away from the object2. Figure Interaction existence classification for the scene "LeftBox" after filtering using [START_REF] Jaffré | Improvement of a Temporal Video Index Produced by an Object Detector[END_REF], shows an interaction began to appear at window starting at frame 40. After analysing all those characteristics, key moments should be extracted according to important variations showing irregularity. The importance of the variations differs from scene type to another and from user needs to another. For example, the object acceleration thresholds depend directly on the scene type, were the velocity of an outdoor circulation scene differs dramatically from an indoor one. Even though in the same indoor type scene, these thresholds differ according to the limitations and conditions set by the responsible on the monitoring system. Another example, the acceleration variations graph presented in the Figure IV-23, belonging to the "LeftBox" scene having a large Depth Of Field (DOF), the user should take that into consideration and select different thresholds to each person, or the key moments of one of them will not be presented in the description. Moreover, while some end users want to keep text generation to the minimum, others want to generate text for every frame.

3-The interaction features

Taking all of that into consideration, and as we want to keep our system generic and context-free, we kept the verbosity of the description density to be controlled by the user according to the scene type. And so, those thresholds can be fixed manually, using a percentage scale or regulator for example, by the end user in order to de determine the amount of text generated by each feature. Different thresholds indicate different number of key moments for each characteristic. Triggered by these key moments, a scene activity characteristics matrix of corresponding characteristics is filled.

In the "LeftBox" scene analysis, as shown in figures

IV.3.8.B. Scene activity characteristics matrix

For each video (VIDEO ID) and each scene (activity ID), which begins from birth frame to death frame, and according to the key moments (𝑘 𝑚 є { k 1 , k 2 , … k n }) extracted, the corresponding characteristics were generated and filled in a vector V(k m ), as follow: 𝑉(𝑘 𝑚 ) = {𝐹𝑟𝑎𝑚𝑒 𝑛𝑏𝑟 (𝑘 𝑚 ), 𝑂𝑏𝑗1 𝑐ℎ𝑎𝑟 (𝑘 𝑚 ), 𝑂𝑏𝑗2 𝑐ℎ𝑎𝑟 (𝑘 𝑚 ), 𝐼𝑛𝑡𝑒𝑟𝑂𝑏𝑗 𝑐ℎ𝑎𝑟 (𝑘 𝑚 ), 𝐼𝑛𝑡𝑒𝑟𝑎𝑐𝑡𝑖𝑜𝑛 𝑐ℎ𝑎𝑟 (𝑘 𝑚 )} (eq. IV -7) Where ObjI char (k m ) = {type, position to an area of interest, position in frame, direction, position vs other object, shape_hu, shape_surface, speed} characteristics at key moment k m , for each object (l=1,2); and InterObj char (k m ) = {distance between objects} characteristic at key moment k m ; and Interaction char (k m ) ={Interaction existence, Interaction type ( Distant (far) vs Physical (close)), Interaction aggressiveness (aggressive vs peaceful)} characteristics at key moment k m .

Finally, all the V(k m ) produce the scene activity characteristics matrix M sac , following the equation: 𝑀 𝑠𝑎𝑐 (𝐴𝐶𝑇𝐼𝑉𝐼𝑇𝑌 𝐼𝐷 , 𝑉𝐼𝐷𝐸𝑂 𝐼𝐷 ) = { 𝑉(𝑘 1 ), 𝑉(𝑘 2 ) … , 𝑉(𝑘 𝑚 ), … 𝑉(𝑘 𝑛 )} (eq. This matrix can be easily used to build sophisticated scenarios models based on context characteristics thresholds, which can be set to generate wanted alerts. Also, this matrix can archive needed characteristics, and make them quickly and easily available using simple queries for later retrieval. An example of M sac can be seen in Table IV-3.

IV.3.8.C. Scene description

For the scene description, we use a template-based method to generate texts to represent objects interactions and activities at the extracted key moments in the scene. For that, we propose new ontology-based (see sub-section II.3.6) generic structured templates containing main information reported by the police in case incident description.

Mainly, we propose two types of templates:

1-Object characteristics templates: for each of the objects, three templates were introduced: a. At the key moment when an object enters the scene, the description follows a template, called "Object entrance template": {"Type" object "ID" enters the scene, {from "Area of Interest" spot | outside areas of interests}, on the "Frame Area Symbol" of the {"Inside" | "Outside"} area of the camera field of view, heading "Direction", [ {"Toward" | "Away from"} the object "ID",] having respectively {"Regular" | "Irregular"} shape, {"Small" | "Medium" | "Big"} Surface, and {"Slow" | "Normal" | "High"} speed}.

NB: In these templates, words between quotes denote value, clauses between curly brackets indicate many clause templates to decide among them according to rules, and clauses between brackets indicate that the clause may not be present in the output sentence (to be decided according to rules).

b. At each of the key moments of an object (other than its entrance or exit), a proposed "Object characteristics template" is structured as follow: {"Type" object "ID" {moves, in c. For an object exiting, a proposed "Object exit template" is structured as follow:

{"Type" object "ID" [leaves "Area of Interest" spot] and exits the scene, {from "Area of Interest" spot | outside areas of interests}, on the "Frame Area Symbol" of the {"Inside" | "Outside"} area of the camera field of view, heading [immediately] "Direction", [ {"Toward" | "Away from"} the object "ID",] {"No big changes occurring respectively on" | "Occurring respectively irregularity in"} its shape, and {"No big changes occurring respectively on" | "big changes occurring respectively on"} its This rule indicates that if the current and last object centroids not belonging to any of the areas of interest we map {moves} into the template.

-For the position and direction compared to the other object: two rules were applied for each of the objects:

As example, for an object 1, at a key moment 𝑘 𝑚 , in a given scene. Let's α be the angle between the vector direction of the object 1 and the vector direction formed by the centroid of object 1 and the centroid of object2:

α = ((𝐶(𝑜𝑏𝑗 1 (𝑘 𝑚-1 )), 𝐶(𝑜𝑏𝑗 1 (𝑘 𝑚 ))) ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ , (𝐶(𝑜𝑏𝑗 1 (𝑘 𝑚 )), 𝐶(𝑜𝑏𝑗 2 (𝑘 𝑚 ))) ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ ) ̂ a) if |α| ≤ 45 °⟹ 𝑎𝑑𝑑 𝑡𝑜 𝑑𝑒𝑠𝑐𝑟𝑖𝑝𝑡𝑖𝑜𝑛 {"𝑡𝑜𝑤𝑎𝑟𝑑" 𝑡ℎ𝑒 𝑜𝑏𝑗𝑒𝑐𝑡 "2"}.
This rule indicates that if the absolute value of α is smaller than 45 degrees, than "object1" is heading toward the "object2". b) if |α| ≥ 135 °⟹ 𝑎𝑑𝑑 𝑡𝑜 𝑑𝑒𝑠𝑐𝑟𝑖𝑝𝑡𝑖𝑜𝑛 {"𝑎𝑤𝑎𝑦 𝑓𝑟𝑜𝑚" 𝑡ℎ𝑒 𝑜𝑏𝑗𝑒𝑐𝑡 "2"} This rule indicates that if the absolute value of α is bigger than 135 degrees, than "object1" is considered going away from the "object2".

Finally, after mapping the scene activity characteristics matrix M sac into the structured templates, the system can generate mainly two kinds of scene textual description, a full description and a short one. The full description describes, for each key moments, all the features values (see Table IV-4), whereas the short one describes only features associated to the most important variations (showing the irregularity) at each moment (see Table IV -5). From the short description, two objects life cycle descriptions can be exported.

An example of describing the scene "LeftBox", taken from the database "Caviar" ("CAVIAR," 2004), containing pictures, key moments, matrixes, full and short description can be seen on the next pages. For another example, the reader can refer to the experiments sub-section IV.4.6.

The scene LeftBox shows a person enters the scene, changes its direction and speed many times, and interacts with another person at distance, see below figures. Next, in Table IV-3, we present the scene activity characteristics matrix Msac showing objects and interaction states for the scene "LeftBox", all the red marked values are key moments corresponding characteristics where irregularity exists. Object 1 shows key moments with its direction when it is near the frame 101, 110, 185, and 230, and with its speed near the frame 160. Object 2 shows keys activities, with its direction when it is near the frame 185, and with its speed near the frame 145, 160 and 230, and with its shape near the frame 230. The interaction between the 2 objects begins near the frame 145 and ends near the frame 190.

In Table IV-4, the full description of the scene "LeftBox", results of mapping the M sac into the proposed templates, is shown. Where, at each key moment, it reflects the corresponding irregularity shown above, and describes the full state of the objects and interaction. To be noticed as example, near the frame 145 the two objects start distant peaceful interaction, which ends near the frame 190.

In Table IV-5, we show the short description of the scene "LeftBox", which describes at each moment only the corresponding irregularity. To be noticed as example, near the frame 145 the two objects start distant peaceful interaction, which ends near the frame 190. Also, the big variations on object 1 directions are when he is near the frame 101, 110, 185, and 230. Table The full description of the scene "LeftBox", results of mapping the M sac into the proposed templates. "Deformable " object "1" moves, in "B" spot, on the "Left Middle " of the "Inside " area of the camera field of view, heading "Up Left", "Toward" the object "2", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having slight "decreasing" of its speed. "Deformable " object "2" exits the scene, from "B" spot, on the "Up Left " of the "Outside " area of the camera field of view, heading "Up Middle", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively slight "decreasing" of its speed.

The two objects are respectively "Approaching", No Interaction occurs between them.

"Deformable " object "1" leaves "B" spot and exits the scene, from "A" spot, on the "Down Right " of the "outside " area of the camera field of view, heading immediately "Down Middle", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively slight "increasing" of its speed.

"Deformable " object "1" moves, in "B" spot, on the "Down Right " of the "Inside " area of the camera field of view, heading "Up Middle", "Toward" the object "2", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively slight "increasing" of its speed. "Deformable " object "2" moves, in "C" spot, on the "Up Left " of the "Outside " area of the camera field of view, heading "Up Middle", "Away from" the object "1", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively "stable" speed.

The two objects are respectively "Approaching", No Interaction occurs between them.

"Deformable " object "1" moves, in "B" spot, on the "Left Middle " of the "Inside " area of the camera field of view, heading immediately "Up Left", "Toward" the object "2", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively slight "decreasing" of its speed. "Deformable " object "2" leaves, "C" spot, and moves, in "B" spot, on the "Up Left " of the "Outside " area of the camera field of view, heading "Up Middle", "Occurring respectively irregularity in" its shape, and "No big changes occuring respectively on" its surface, and having respectively considerable "increasing" of its speed.

The two objects are respectively "Approaching", No Interaction occurs between them.

The two objects are respectively "Approaching", A "Distant" "Peaceful" Interaction occurs between them.

"Deformable " object "1" moves, in "B" spot, on the "Up Left " of the "Inside " area of the camera field of view, heading immediately "Down Right ", "Away from" the object "2", "No big changes occurring respectively on" its shape , and "No big changes occurring respectively on" its surface, and having respectively "stable" speed. "Deformable " object "2" moves, in "B" spot, on the "Up Left " of the "Outside " area of the camera field of view, heading "Down Middle", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its Surface, and having respectively "stable" Speed.

The two objects are respectively "Receding", no Interaction occurs between them.

"Deformable " object "1" leaves, "B" spot, and moves, in "A" spot, on the "Down Middle " of the "Inside " area of the camera field of view, heading "Down Right ", "Away from" the object "2", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively slight "increasing" of its speed. "Deformable " object "2" moves, in "B" spot, on the "Up Left " of the "Outside " area of the camera field of view, heading "Down Middle", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively considerable "decreasing" of its speed.

The two objects are respectively "Receding", A "Distant" "Peaceful" Interaction occurs between them.

"Deformable " object "1" leaves, "A" spot, and moves, in "B" spot, on the "Down Right " of the "Inside " area of the camera field of view, heading "Down Right ", "Away from" the object "2", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively considerable "decreasing" of its speed. "Deformable " object "2" leaves, "B" spot, and moves, in "C" spot, on the "Up Left " of the "Outside " area of the camera field of view, heading "Down Middle", "Toward" the object "1", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively considerable "increasing" of its Speed.

The two objects are respectively "Receding", A "Distant" "Peaceful" Interaction occurs between them.

"Deformable " object "1" moves, in "B" spot, on the "Down Right " of the "Inside " area of the camera field of view, heading immediately "Up Middle", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively slight "increasing" of its speed. "Deformable " object "2" moves, in "C" spot, on the "Up Left " of the "Outside " area of the camera field of view, heading immediately "Up Middle", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively slight "decreasing" of its speed.

"Deformable " object "1" enters the scene, from " A " spot, on the "Down Middle " of the "Outside " area of the camera field of view, heading "Up Middle ", having respectively "Regular " shape, "Small " surface, and "High " speed. "Deformable " object "1" leaves, "A" spot, and moves, in "B" spot, on the "Up Left " of the "Inside " area of the camera field of view, heading immediately "Right Middle ", "No big changes occurring respectively on" its shape, and "No big changes occurring respectively on" its surface, and having respectively slight "decreasing" of its speed. "Deformable " object "2" enters the scene, from " B " spot, on the "Up Left" of the "Outside " area of the camera field of view, heading "Down Middle ", having respectively "regular " shape, "small " surface, and "low " speed.

The two objects are respectively "Far", No Interaction occurs between them.

Textual description

Extracted from the short description described above, object 1 and object 2 life cycles can be described as below:  Object1 life cycle description:

-Near the frame 11, deformable object 1 enters the scene, from A spot, on the down middle of the outside area of the camera field of view, heading up middle, having respectively regular shape, small surface, and high speed.

-Near the frame 101, deformable object 1 leaves, A spot, and moves, in B spot, on the up left of the inside area of the camera field of view, heading immediately right middle. -Near the frame 110, deformable object 1 heading immediately down right, away from the object 2. -Near the frame 145, deformable object 1 leaves, B spot, and moves, in A spot, on the down middle of the inside area of the camera field of view, away from the object 2, and starts a distant peaceful interaction with object 2. -Near the frame 160, deformable object 1 leaves, A spot, and moves, in B spot, on the down right of the inside area of the camera field of view, away from the object 2, and having respectively considerable decreasing of its speed. -Near the frame 185, deformable object 1 heading immediately up middle.

-Near the frame 190, deformable object 1 moves, toward the object 2, and no more interaction occurs with object 2. -Near the frame 230, deformable object 1 moves, on the left middle of the inside area of the camera field of view, heading immediately up left, toward the object 2. -Near the frame 238, deformable object 1 moves toward the object 2.

-Near the frame 412, deformable object 1 leaves B spot and exits the scene, from A spot, on the down right of the outside area of the camera field of view, heading immediately down middle.

 Object2 life cycle description:

-Near the frame 101, deformable object 2 enters the scene, from B spot, on the up left of the outside area of the camera field of view, heading down middle, having respectively regular shape, small surface, and low speed. -Near the frame 145, deformable object 2 having respectively considerable decreasing of its speed, start a distant peaceful interaction with object 1. -Near the frame 160, deformable object 2 leaves, B spot, and moves, in C spot, toward the object 1, and having respectively considerable increasing of its speed. -Near the frame 185, deformable object 2 heading immediately up middle.

-Near the frame 190, deformable object 2 moves, away from the object 1, no more interaction occurs with object 1. -Near the frame 230, deformable object 2 leaves, C spot, and moves, in B spot, occurring respectively irregularity in its shape, and having respectively considerable increasing of its speed. -Near the frame 238, deformable object 2 exits the scene, from B spot.

IV.4. Experiments and results

As the proposed method consists in many stages, we made experiments to evaluate each step of the following process:

-Selection of the datasets.

-Object tracking and segmentation.

-Features extraction on each window from scenes having two objects, and data preprocessing. -Classifications using three deep neural network algorithms:

1) The first algorithm was learned and tested to identify, the existing of the interactions in those scenes.

2) The second algorithm was learned and tested to classify, the interactions when exists, if they are distant or physical interaction in those scenes.

3) The third algorithm was learned and tested to classify, the interactions when exists, if they are aggressive or peaceful interaction in those scenes. -As an example scene, the scene activity characteristics matrix M sac was calculated, and then mapped into templates to generate textual descriptions.

In the following we address each of these steps.

IV.4.1. Datasets selection

Deep NNs are requiring a huge amount of training data. However, what we seek for in the scene is rare. I fact, we want annotated video datasets with descriptions about far, physical, aggressive or peaceful interactions between only two objects. For that, a huge effort was made to gather publicly available datasets concerning videos objects actions and interactions (see Table IV-6), including some diversity about object types and area of application. Then, from all these datasets, 323 scenes of 1903 seconds, the ones having only two objects were extracted and manually annotated, as for, existence of interaction or not, physical or far interaction, peaceful or aggressive interaction. Some extracted scenes were not taken into consideration as the tracking and segmentation algorithm missed to detect the two objects. Consequently, no acceptable features could be extracted for further interpretation. After that, the sixth and seventh stages, mentioned in section IV.3., were applied including features extractions from sliding windows of frames, see subsection IV.4.3, then, the data was prepared and pre-processed, including the dataset increasing by reversing the footage and other methods, see sub-section IV.4.4. Finally, 285 scenes were studied.

Next, according the extracted scenes and the annotations, the three classifiers were trained and tested. Later, a temporal-consistency analysis was applied, to filter the 3 sequential classification vectors, which were used for description.

Nb: for more information about the videos existing in these datasets, the reader can refer to [START_REF] Chaquet | A survey of video datasets for human action and activity recognition[END_REF], ("CV Datasets," n.d.), ("YACVID," 2018), [START_REF] Borges | Video-Based Human Behavior Understanding: A Survey[END_REF]. Collective Activity Dataset [START_REF] Choi | A Unified Framework for Multi-target Tracking and Collective Activity Recognition[END_REF] This dataset consists of many sets of images concerning mainly of people 'Crossing', 'Waiting', 'Queuing', 'Walking', 'Running', and 'Shaking hands'. Based on moving objects detection by background subtraction algorithm; then, on Kalman filter for tracking and predicting the moving objects from frame to frame.

Easy +++

IV.4.2. Segmentation and tracking algorithms comparison

Many efforts have been done in this field by the research community. We searched for available multi-target/object tracking and segmentation algorithms, seeking for a simple one, easy to install, that can provide us with acceptable results for further interpretation. After an exhaustive search for available algorithm in the domain of segmentation and tracking, more than 20 algorithms were found. We list in the Table IV-7 above 10 of the tested algorithms and their ability to fulfil the properties we are looking for. Some published algorithms were not available as functional code; others deliver the tracking data without the segmentation results (Pirsiavash, n.d.) or the segmentation results without the tracking ones; others did not handle the propriety of multi-object tracking. On the other hand, we found codes compliant with our conditions, but only adapted for humans [START_REF] Jiang | Multi-person Tracking-by-Detection Based on Calibrated Multi-camera Systems[END_REF], [START_REF] Benfold | Stable multi-target tracking in real-time surveillance video[END_REF], [START_REF] Choi | A Unified Framework for Multi-target Tracking and Collective Activity Recognition[END_REF], [START_REF] Possegger | Occlusion Geodesics for Online Multi-object Tracking[END_REF], [START_REF] Milan | Discrete-Continuous Energy Minimization for Multi-Target Tracking[END_REF], [START_REF] Milan | Continuous Energy Minimization for Multi-Target Tracking[END_REF], [START_REF] Fragkiadaki | Two-Granularity Tracking: Mediating Trajectory and Detection Graphs for Tracking under Occlusions[END_REF], [START_REF] Roshan Zamir | GMCP-Tracker: Global Multi-object Tracking Using Generalized Minimum Clique Graphs[END_REF].

After all, we choose the algorithm for segmentation and multi-object tracking provided by Matlab ("Motion-Based Multiple Object Tracking -MATLAB & Simulink," n.d.), as it is the most suitable in our case. It is multi-object tracker with object segmentation, which is easy to implement, and have acceptable results.

IV.4.3. Data preparation and pre-processing

We searched scenes having only two moving objects taken from fixed camera in all the above-mentioned datasets. Some of those scenes contain interaction between the two objects and others do not. As not many works were done in the case of distant interaction, and its aggressiveness, we did not find in these datasets a large diversity of scenes. Finally, after creating sliding window, for our experiments we have chosen the size to have 25 frames, we extracted 2498 features from each window. Five types of features were mined, spatial, temporal, inter-objects, inter-frames and trajectory features. We found in all those datasets around 6029 windows, from which only 2208 windows of interaction. Between the 2208 interactions windows, we were able to differentiate 5962 distant interactions, and only 67 physical interactions.

Despite that the used segmentation and tracking algorithm ("Motion-Based Multiple Object Tracking") deals with the common occlusion problem by predicting and correcting its location using "Kalman filter", the results still suffer from this problem when two objects physically interact, because interaction may occur for a long time. After a given number of frames (8 frames in this case), one of the objects is lost. Both objects are labelled as being one. After that the object finishes interacting, the algorithm detects and labels one of the objects as being a different object than before the interaction. Hence, in the case of physical interaction or when there is close distance between objects, the windows automatically selected and showing two different objects are in a limited number. Consequently, we implement the whole procedure for all the dataset after reversing the video to increase the number of available samples for learning (especially for the case the physical interaction), and to catch the last moments of interaction between the objects.

Increasing artificially datasets is a typical approach, especially when the number of positive examples is too small. Here the reversed footage they were judged and annotated as it were played in the regular temporal order. The reversed video does not change the classification statement of interaction, if it exists, neither for distant and physical interaction, nor for aggressive and peaceful interaction. Also, the features were trimmed to optimize the results of classifications. As in some cases, some values cannot be computed; we removed some of the features with missing values. Mainly those features are first and second derivative of a characteristic like Hu moments, or distance between objects, etc. As results, we kept only 2305 features.

Finally, we were able to have around 11200 windows taken from 285 scenes with duration of scenes taken and tested are shown in Table IV-6. From these 11200 windows only 3955 are representing interaction. Between the 3955 interactions windows, we were able to differentiate 3692 distant interactions, and 263 physical interactions, 438 aggressive and 3517 peaceful interaction windows. Despite of the size of collected multiple datasets; the global amount of data remains insufficient for sharp evaluation benchmark. After preparing the input dataset for the three classification algorithms, we suffered from the imbalance between the negative and positive inputs for the three classifications. For better classification results, we choose to duplicate the number of positive results.

At the end, for each of the three classification algorithms the balanced dataset input ends up to be like the following in the

IV.4.4. Classification training and results

According to the three corresponding balanced datasets, mentioned above, several machine learning algorithms were tested to determine, the existence of interactions, and if they are distant or physical, and if they are aggressive or peaceful.

For each of the three classifications, in the MATLAB and Simulink environment, we used "Classification Learner app" ("Classification Learner App -MATLAB & Simulink," n.d.) from the "Statistics and Machine Learning Toolbox" to train and test different models of several classical machine learning algorithms, and "Deep Learning Toolbox" ("Deep Learning Toolbox Matlab," n.d.) to implement, train and test a multi-layered Deep Neural Network DNN, as follow: -For the classical machine learning algorithms experimentations: using the Classification Learner app, we performed automated training to search for the best classification model type, including decision trees (simple, medium, and complex), discriminant analysis (linear and quadratic), logistic regression, support vector machines (linear, quadratic, cubic, fine Gaussian, medium Gaussian, and coarse Gaussian), nearest neighbors (fine KNN, medium KNN, coarse KNN, cosine KNN, cubic KNN, and weighted KNN), and ensemble classifiers (boosted trees, bagged trees, subspace discriminant, subspace KNN, and RUSBoost trees). Then, for the classification models showing best results, many trainings were performed after fine tuning the corresponding parameters to maximize the results. Finally, for each of the three classifications, one classification DNN model, showing ultimate results, were selected.

In the following, we present, for each of the three classifications after experiments and testing, the best results using a classical machine learning algorithm and the best results using a DNN:

1-Interaction vs non-interaction classification:

For the classical machine learning algorithm classification, we used as input dataset 887 non-overlapping windows records (224 interactions and 663 non-interactions). The algorithm that shows best results after many tests was "AdaBoost decision Tree" from the ensemble methods with 82% of accuracy, having maximum number of splits (tree depth) set to 30 and the number of learners set to 50, see Figure IV-30.

For the DNN, we used the corresponding balanced data mentioned in the Table IV-8 above (14902 input records having each 2305 features, from which 7657 classified as interaction and 7245 classified as non-interaction), and we chose completely different scenes for each of the training, validation and test sets. The chosen algorithm, as mentioned in sub-section IV.3.7.A, is the seven-layer Pattern recognition network, a feedforward network composed of fully connected layers, six hidden layers and one output layer. Each hidden layer contains 586 neurons. As a method for network learning we used the "trainscg", which is an implementation of the scaled conjugate gradient backpropagation method, the six hidden layers activation function is the transfer function Log-sigmoid "logsig" and a softmax transfer function in the output layer. Other parameters are shown in Table IV-9 For the classical machine learning algorithm classification, we used as input dataset 239 non-overlapping windows records (13 physical and 226 distant). The algorithm that shows best results after many tests was "Bag Decision Tree" from the ensemble methods with 89.5% of accuracy, having maximum number of splits (tree depth) set to 30 and the number of learners set to 30.

For the DNN, we used the corresponding balanced data mentioned above (7079 input records having each 2303 features, from which 3640 classified as physical interaction and 3439 classified as distant interaction), and we chose completely different scenes for each of the training, validation and test sets. The chosen algorithm is the four-layer Pattern recognition network, a feedforward network composed of fully connected layers, three hidden layers and one output layer. Each hidden layer contains 426 neurons. As a method for network learning we used the "trainscg", which is an implementation of the scaled conjugate gradient backpropagation method, the three hidden layers activation function is the transfer function Log-sigmoid "logsig" and a softmax transfer function in the output layer. Other parameters are shown in Table IV-9. As result of training this algorithm and testing it with 7079 records and 2303 features, having a training set of approximate 70% and a validation and test sets approximate 15 % each one, the accuracy of the test set achieved 93.7% after 102 epochs.

3-Aggressive and peaceful interaction classification:

For the classical machine learning algorithm classification, we used as input dataset 239 non-overlapping windows records (24 aggressive and 215 peaceful). The algorithm that shows best results after many tests was "Bag Decision Tree" from the ensemble methods with 87.9% of accuracy, having maximum number of splits (tree depth) set to 20 and the number of learners set to 30.

For the DNN, we used the corresponding balanced data mentioned above (6703 input records having each 2303 features, from which 3439 classified as aggressive and 3264 classified as peaceful), and we chose completely different scenes for each of the training, validation and test sets. The chosen algorithm is the four-layer Pattern recognition network, a feedforward network composed of fully connected layers, three hidden layers and one output layer. Each hidden layer contains 546 neurons. As a method for network learning we used the "trainscg", which is an implementation of the scaled conjugate gradient backpropagation method, the three hidden layers activation function is the function Logsigmoid "logsig" transfer function and a softmax transfer function in the output layer. Other parameters are shown in Table IV-9. As result of training this algorithm and testing it with 6703 records and 2303 features, having a training set of approximate 70% and a validation and test sets approximate 15 % each one, the accuracy of the test set achieved 93.8% after 88 epochs.

These experimental results show that the classic machine learning algorithms (AdaBoost, and Bag Trees) results are quite comparable to the DNNs results, with a small advantage to the latter ones.

IV.4.5. Scenes description results

In addition to the scene described in section IV.3.8, where it is mainly focused on different variation of object characteristics and peaceful interaction between the two objects; using the same methodologies, we present here the description another scene named "Fight_RunAway2" taken from the database "CAVIAR" ("CAVIAR," 2004). This scene shows a fight between two persons, where the main focus is the detection of distant aggressive interaction, and to show the effect of the tracking and segmentation algorithm false detection.

To better understand the scene, important moments are shown in the following figures: The graph-based pattern discovery was implemented and the new scene activity characteristics matrix was filled to highlight appropriate key moments for description phase. As mentioned, we kept the verbosity of the description density to be controlled by the user; next we select thresholds in a way to show only the major irregularities. These characteristics are:

3-The interaction features:

In this example, and because of the occlusion caused by the physical approach of the two objects, there is a false detection, by the used tracking and segmentation algorithm, between the frame 290 and the frame 426. Thus, the algorithm detects them as one big object. As a result, the system was not able to detect the physical aggressive interaction that starts at frame 321 and ends at frame 397. And so, only the start of distant aggressive interaction was detected at frame 279, and its end at frame 467.

From the extracted key moments (km), the corresponding characteristics were generated and filled in the vector V(k m ), to produce the scene activity characteristics matrix M sac . Finally, the short description is shown in Table IV-10.

To be noticed in the description, the two objects after entering the scene, start approaching toward each other and decreasing their speeds. Then, near the frame 279 the two objects start distant aggressive interaction.

Between the frames 290 and 320, one of the objects was occluded by the other, and then a physical interaction starts near the frame 321 till the frame 397, followed by another occlusion till the frame 426. Consequently, as the system only pre-processes, for classifications, the interaction between two objects, and as the two objects were false detected by the tracking and segmentation algorithm as being one, the system was unable to generates description between the frame 291 and the frame 425.

Later in the description, one can easily note the distant peaceful interaction between the frames 426 and 468, while the two persons were rolling away, before exiting the scene. "Deformable" object "1" having respectively slight "decreasing" of its Speed. "Deformable" object "2" heading "Down Left ", having respectively slight "increasing" of its Speed. The two objects are respectively "Approaching", no more interaction occurs between them.

476

"Deformable" object "1" heading "Up Middle ", "Toward" the object "2", "Occurring respectively irregularity in" its shape, having respectively slight "decreasing" of its Speed. "Deformable" object "2" heading "Up Left ", "Toward" the object "1", having respectively slight "decreasing" of its Speed. "Deformable" object "1" exits the scene, from "A" spot, heading "Toward" the object "2", "Occurring respectively irregularity in " its shape. "Deformable" object "2" exits the scene, from "A" spot.

"Deformable" object "1" moves, "Away from" the object "2", "Occurring respectively irregularity in " its shape, and having respectively slight "increasing" of its Speed. "Deformable" object "2" moves, "Away from" the object "1", having respectively slight "decreasing" of its Speed.

452

"Deformable" object "1" moves "Away from" the object "2", having respectively slight "decreasing" of its Speed.

42

"Deformable" object "2" heading immediatly "Up Left ", "Away from" the object "1", having respectively slight "decreasing" of its Speed.

462

"Deformable" object "1" heading immediatly "Left Middle ", having respectively slight "increasing" of its Speed. "Deformable" object "2" moves "Away from" the object "1", having respectively slight "decreasing" of its Speed.

153 426 "Deformable" object "1", leaves "B" spot, heading immediatly "Down Left ", "Occurring respectively irregularity in" its shape, and "big changes occuring respectively on" its Surface having now "Smaller" one, and having respectively considerable "increasing" of its Speed. "Deformable" object "2", leaves "B" spot, heading immediatly "Up Middle ", "Occurring respectively irregularity in" its shape, and "big changes occuring respectively on" its Surface having now "Smaller" one, and having respectively considerable "increasing" of its Speed.

ACTIVITY NUMBER

The two objects are respectively "Receding", A "Distant" "Peaceful" Interaction occurs between them.

429

"Deformable" object "1" moves, in "A" spot, "Away from" the object "2", having respectively considerable "decreasing" of its Speed. "Deformable" object "2" moves, in "A" spot, on the "Up Left " of the "Outside " area of the camera field of view, "Away from" the object "1". 447 290 "Deformable" object "1" heading immediatly "Right Middle ", "Occurring respectively irregularity in " its shape , and "big changes occuring respectively on" its Surface having now "Bigger" one, and having respectively considerable "decreasing" of its Speed. "Deformable" object "2" heading immediatly "Up Middle " , "Toward" the object "1", "Occurring respectively irregularity in " its shape, and "big changes occuring respectively on" its Surface having now "Bigger" one, and having respectively considerable "increasing" of its Speed.

Birth

Frame 279 "Deformable" object "1" having respectively slight "decreasing" of its Speed.

"Deformable" object "2" heading "Left middle " , "Toward" the object "1", having respectively considerable "decreasing" of its Speed. "Start" a "Distant" "Agressive" Interaction.

1

"Deformable" object "2" leaves "A" spot, and moves "Toward" the object "1", "big changes occuring respectively on" its Surface having now "Smaller" one, and having respectively slight "decreasing" of its Speed.

263

"Deformable" object "1" heading "Up Right ", having respectively slight "increasing" of its Speed. "Deformable" object "2" moves, in "B" spot, on the "Left Middle " of the "Inside " area of the camera field of view, and having respectively slight "decreasing" of its Speed.

232

"Deformable" object "1", leaves "A" spot, having respectively slight "increasing" of its speed. "Deformable" object "2", moves, on the "Down Middle " of the "Inside " area of the camera field of view, "Toward" the object "1", and having respectively considerable "Increasing" of its speed.

243

"Deformable" object "1" moves, in "B" spot, heading immediately "Up Middle ", and having respectively slight "decreasing" of its speed. "Deformable" object "2" moves on the "Down Left " of the "Inside " area of the camera field of view, "Toward" the object "1", and having respectively considerable "Increasing" of its speed.

"Deformable" object "2" moves, on the "Down Middle " of the "Outside " area of the camera field of view. The two objects are respectively "Approaching".

Death

Frame "Deformable" object "2" enters the scene, from " A " spot, on the "Down Right " of the "Outside " area of the camera field of view, heading "Up Left ", having respectively "regular " shape, "medium " surface, and "Normal " speed. The two objects are respectively "far".

210

"Deformable"object "1" heading "Up Right ", "Occurring respectively irregularity in" its shape , and having respectively slight "decreasing" of its speed.

Textual description 551

"Deformable " object "1" enters the scene, from " A " spot, on the "Left Middle " of the "Outside " area of the camera field of view, heading "Up Left ", having respectively "regular " shape, "small " surface, and "slow " speed.

"Deformable" object "1" Occurring respectively irregularity in its shape.

202

"Deformable"object "1" having respectively slight "Increasing" of its speed.

IV.5. Discussion

After running all the tests on our system, some difficulties show up, especially with the segmentation and tracking algorithm results where it suffered from one major traditional issue and another marginal one:

1-The major issue: is the traditional occlusion when two moving objects are physically close. Using Kalman filter in the tracking algorithm to estimate the location when an object is occluded worked well on occlusion with an inert object but not with a moving one. In latter case the occluded object location and boundary box were estimated for some number of frames (the default is 8 frames), while the foreground object is missdetected by detecting both objects as one. This problem was a major one because its results affect all the system, as seen in the example in section IV.4.6. Consequently, the physical interaction in a scene was detected for only 8 frames per scene. And then, analysing and describing the interaction had no more effect until the two objects separate. 2-The marginal issue: is the false segmentation of the object when it is moving in a complex background (illumination, and high texture). This issue can trigger a description declaring a big change in the object Hu moments or surface, as the example seen in section description IV.3.8. This can be over passed at the level of thresholding.

For these particular problems caused by the tracking and segmentation algorithm, as our system is flexible, with a simple "plug and play" this algorithm can be replaced. Having lately good results with detecting objects using deep learning, like YOLOv2 and Mask R-CNN, a good plan could be by testing these algorithms and applying a pre-processing stage to extract the objects segmentation, then if one of them deliver better results and satisfy all the conditions, our selected algorithm can be replaced.

The detection of interaction existence tests show 12.5% of false detection, where 11% are due to false detection of non-interactions as interactions. However, detecting at which frame a far interaction starts it is very delicate, even for human brain, where sometimes it seems more subjective. Two experts can identify, according to their perspectives, the start of far interaction at different frames.

While the classification of aggressiveness is more subjective with hardly acceptable 6% of false classification, differentiating between physical and distant interaction should be accurate. Nonetheless, the 6 % of false classifications between distant and physical are mainly due to the false detection, by the tracking and segmentation algorithm, of the two objects when approaching as being one even before the physical contact.

Despite of the size of collected multiple datasets, the global amount of extracted data remains insufficient for sharper evaluation benchmark, and this affect directly the results of the classification algorithms. This needs to be improved by testing this system on real footage of surveillance, were having more data can improve dramatically the classification accuracy.

Moreover, it is difficult to evaluate video description, some systems typically perform an automatic quantitative evaluation of their descriptive sentences using machine translation and image captioning metrics such as BLEU [START_REF] Papineni | BLEU: A Method for Automatic Evaluation of Machine Translation[END_REF], ROUGE [START_REF] Lin | ROUGE: A Package for Automatic Evaluation of Summaries[END_REF], METEOR [START_REF] Banerjee | METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments[END_REF], SPICE [START_REF] Alonso | SPICE: Semantic Propositional Image Caption Evaluation[END_REF] and other metrics. But assessing how good the semantic representation of visual content is, it is not a straightforward task. A video can be correctly well described in a variety of sentences. Similarly, if many persons where asked to describe the same scene, they can provide different descriptions, each one from his own perspective. This indicates that video description is also subjective and uncertain. Beside this, in many practical cases, human activities are too complex to be described with short, simple sentences, in only one way. For that, as the video description templates and models are abstractions of the natural video description processes, they had to focus only on the relevant and prominent components, thus models can be diverse and uncertainty rises. [START_REF] Song | From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning[END_REF]. And so, evaluating video description is a hard task, either automatically generated or manually, because there is no absolutely correct answer, and no absolute standards to measure systems outputs. For this end, for correct evaluations, many systems provide a human expert assessment instead of the automated ones [START_REF] Awad | TRECVID 2018: Benchmarking Video Activity Detection, Video Captioning and Matching, Video Storytelling Linking and Video Search[END_REF], [START_REF] Graham | Evaluation of automatic video captioning using direct assessment[END_REF], [START_REF] Graham | Evaluation of automatic video captioning using direct assessment[END_REF], [START_REF] Aafaq | Video Description: A Survey of Methods, Datasets and Evaluation Metrics[END_REF], [START_REF] Ahmed | Natural Language Description of Surveillance Events[END_REF]. For our approach, we found this stage irrelevant as our description output will be according to the structured templates, reflecting our reports needs as experts in the practical field.

On the other hand, our approach provides many contributions, we mention: -In our approach, the input features are dedicated to the interaction classification process, where many of other methods do not export appropriate features from the videos. -Our approach took into consideration the diversity of scenes, context, objects type, actions and interactions, where no conditions and restrictions were applied. -Our approach is benefitting from machine learning and DNN, in its experimental phases.

-Our approach implements the original new classification idea of distant vs physical interaction, while other works focus only on the physical one. Moreover, detecting distant aggressive interaction can alert the surveillance control rooms' observers at early stages, giving them precious time to act. -The experimental results show, in our classification fields, that the classic machine learning algorithms (AdaBoost, and Bag Trees) can produce quite comparable results to the DNNs, with a small advantage to the latter ones. -The used features for description can be extended to add object colour and other features, and at the same time be used for querying the data, which is a great need in CCTV systems. All the above-mentioned extracted features are expressed and stored in the database, under the scene activity characteristics matrix M sac , as a high-level symbolic description of the object's activity. This metadata contains information driven by time for each of the detected objects and interactions including: trajectory and routes taken through the field of view, time of interaction, speed and its variation, shape and its changes, directions and its variations, deformability and interaction with other objects, aggressiveness or not for its interaction, and others. This information is attached to each object detected by the system. Such intermediate information may be very helpful to the end user, especially the operators as it can be set up to generate alerts, and the investigators when searching the archive for an incident with specific description. We tried to surround, in those extracted features, most of the queries used in practice to search for an incident. For example: we can search for a car coming from the north east, heading to the south west, speeding and had an accident, by searching in M sac for big deformable object coming from the upper right of the outside area of the camera FOV, heading down left, toward the other object, having respectively considerable increasing in its speed, and both objects are respectively approaching. -Our approach present a new ontology-based generic non-contextual way for description using well-structured generic templates, compared to the ones found in the state-ofthe-art like [START_REF] Ahmed | Natural Language Description of Surveillance Events[END_REF], see sections IV.2.5 and IV.3.8. -In our approach, we are not competing with the state-of-the-art video description frameworks. Our video description approach is a well-designed framework which focuses, mainly, on examination of interaction analysis, understanding and description for video surveillance system. It is, only, one more step forward, toward an advanced level of intelligent surveillance system. -In our approach, the form of the textual description is controlled by predefined wellstructured templates, not textual descriptions that are training the system. The verbosity of the description can be tuned at any time by the end user. -The new structured templates containing main information reported by the police in real case incident description. Consequently, the output textual description can be generated automatically as draft reports to be based on.

IV.6. Conclusion

In this chapter, we presented a new generic non-contextual approach, based on the ontology seen in chapter II, for description video surveillance scenes. A new set of features, appropriate for videos, was extracted from the scenes after applying an off-shelf tracking and segmentation algorithm. A new classification of scenes types was proposed, based on background changes, the number of objects and variations of object characteristics. Interactions types classifications, based on DNN, take role. A graph-based pattern discovery was implemented and new matrix of scene activity characteristics was introduced to highlight appropriate key moments for description phase. Finally, new rule-based templates were proposed to structure the textual descriptions of the scenes. The verbosity of these descriptions can be controlled by the user.

V. Surveillance systems -Between theory and practice

V.1. Introduction

The law enforcement agencies are in constant search for effective public safety and security strategies to help deal with criminal and terrorist acts. While many believe that the adoption of "community policing" strategy has led to greater efficiency and effectiveness in policing, law enforcement agencies are interested in using new and effective tools that can enhance these community policing efforts, particularly in public places.

Among the most modern public safety and law enforcement tools adopted is the use of surveillance systems, in particular, to combat crime and terrorist acts in public places. Law enforcement agencies believe that community policing, which embodies a combination of proactive crime prevention and community involvement with more traditional policing functions, can be used; since monitoring public places can promote problem solving strategies, assist in arrests and investigations, and increase the fear of criminals from the possibility of arrest. In addition, it may be considered that camera surveillance systems in public places may also have positive effects, such as increasing users' sense of public safety and improve partnership between the community and the Police against crime.

For more than three decades, camera surveillance systems have been used in many countries including Britain, France, Spain, the United States, Monaco and others. After September 11, 2001 attacks, the use of these systems became more common in order to deter future terrorist attacks. This incident highlighted the contribution of large surveillance systems to crime detection and prevention, leading to further improvements and developments in these systems.

The use of such a system in the field of safety and security varies from system to another according to the system goals and context. For example, it can be used in public places for crime fighting and order maintenance, or it can be used in critical facilities like airports or seaports and for security reason and threats detection, or on border for intrusions detection or in metro stations for security and safety, etc. Also, the use is not restricted to security, but it can trespass it for many other fields as nursing like for elderlies or enfant, management like for traffic or industries or sports fields, statistics like in sports or merchandise, etc.

Due to the huge number of videos produced from the surveillance systems cameras, two main problems face the surveillance control room management and operators, first is the shortage of active monitoring associated with the need of automatic alerts, and the second is the difficulties of investigating the archives.

As mentioned, enormous work has been done on the analysis and understanding of videos in general, and the surveillance systems in particular; several significant solutions from the research and market fields have been proposed. However, the lack of generic and precise models for video content representation, and the high complexity and diversity of video scenes, make building of fully automated intelligent video analysis and description a challenging task. Additionally to these difficulties facing these automated systems, and because of the diversity of end users needed outputs, the application domain still shows a big gap between what is needed by the end users, what is produced in the research field and what is delivered by the companies as video analytics tools. By end user, we mean law enforcement personnel, security officer, or any surveillance system operator.

The work presented here was developed with "Beirut CCTV control room". This last chapter of contribution is aiming at confronting our experience in the field of academic research and our expectations in the field of actual management of video surveillance systems. The main intent of it is to highlights the existing gaps, and to give scientists, engineers and managers alike, a general understanding, from the theoretical and practical perspectives, of the available solutions and practical needs, involved with the surveillance system. To do so, we take some distance with the precise scope of our previous contributions to enlarge our field of view to all the main aspects of contributions and expectations in this area before concluding this thesis.

In the following section V.2, we present a quick surveillance system overview about where and why it is used, as well as the existing video analytics in the market. Later, in section V.3, we explain how surveillance systems can be used to fight the crime. In the section V.4, we highlight some existing gaps between practice and research field as well as our propositions on how to reduce some of these gaps.

V.2. Surveillance system overview

Visual surveillance systems are one of the most important surveillance systems used and the most effective weapon to combat crime and terrorism in public places.

These systems are now a dominant feature of institutional security systems and are used in multiple locations and areas, and for different purposes. Places where surveillance systems are used by the public or private sector are: -Public places: streets and intersections, especially in city centres, squares and parks, shopping centres, museums and public libraries. -Public utilities like the airports and seaports, military and security facilities, prisons and detections, and transportation (trains, tramways, metro, buses, cars, planes, etc.). -Inside and surrounding buildings, public and private institutions, industrial buildings, environmental places and embassies. -Inside and surrounding sports venues, health centres, hospitals, pharmacies, hotels, banks, ATMs, restaurants, schools and universities.

Camera surveillance systems can be used in almost all aspects of life and are an important tool for management, security and law enforcement. These systems can therefore be used in vast fields, and for different purposes as needed. We mention: -Monitoring and managing public and private establishments.

-Traffic management.

-Control and management of crowds. -In the scientific field, military and space research.

-For public safety, security and law enforcement: where the purposes and objectives are to fighting crimes and terrorist acts, protection of property and individuals, assistance the law enforcement agencies in decision-making operations (street demonstrations, celebrations, official VIP movements, emergencies, etc.), risk management in fire, natural disaster and crime situations, security monitoring of sensitive places, collecting public information, and laws enforcement (such as traffic law).

Many commercial system providers exists which offer surveillance video analytics solutions for residential, commercial, and law enforcement agencies. Next we mention, most known deliveries from different video analytics tools existing in the market: -Motion detection -Automatic number plate recognition ANPR system, also known as License Plate Recognition (LPR). -Facial and iris recognition -People-counting -Abandoned or Removed Object Detection -Intrusion or Trespassing, also known as Perimeter Protection -Vandalism and camera tampering detection -Detecting and counting crowds -Loitering -Object tracking: where they provide, according to this, stationary vehicle, direction detection, wrong way detection, illegal turn. -Vehicle type recognition -Colour detection -Pedestrian detection, on highway.

-Gait analysis -Congestion and accident detection -Smoke/fire detection Also, some providers offer investigation tools where you can search the database according to some features, mainly speed, colour, and type (pedestrians or car). To our knowledge there are no frameworks allowing to evaluate and to compare technologies integrated by the providers in their systems. Furthermore, there is a total lack of certification for such precise software pieces that are dealing with citizen's security. Therefore, there is a necessity for an independent evaluation of such capabilities. The most used and reliable ones, comparing to others, are the ANPR, and facial recognition.

V.3. Surveillance systems for fighting crime and terrorist acts

Anti-social behaviour in public places has significant costs on the society, the economy, and the lives of citizens. For example, in Beirut, from the beginning of year 2019 till the end of March, more than 970 crimes are occurring in public places risking citizens' lives, from which, for example, 175 are snatching. Another example, in UK, from 2007 till now, graffiti and vandalism alone cost to the British government around £ 3.4 billion pounds sterling a year.

From this perspective, all countries are searching for a valuable tool in the field of public safety and security management, and in the protection of persons and property, especially in the fight against crime and terrorist acts by preventing them. Many means, beside surveillance systems, are used; we mention streets lights, or false cameras (we mean by these cameras covers). After experience, law enforcement agencies are adopting the visual surveillance systems, and pushing to more improvement to answer all their needs and goals.

All cameras in any surveillance system are sending their signals to be stored in archive. Therefore, two methods of system usage can be mainly distinguished:

1-Archive search: by the 'investigators', and because of the high technicality and difficulty, investigators must be well trained, to harness the technology and use it efficiently. 2-Active Monitoring: means the interactive surveillance by observers to try to detect incidents and monitor them during their occurrence. Detecting an incident during its occurrence is a very difficult task, especially when the operator has to deal with a large number of cameras and a wide geographical area.

In both cases, the video analytics can play a key role in optimizing the system usage.

How surveillance systems fight the crimes? Surveillance systems lead to a major shift in the fight against crime and terrorist acts, along two main ways: indirectly and directly. All this may call for a change in the organization of the police and its way of policing in the face of crime:

V.3.1. The indirect effect

Surveillance systems have indirect, non-negligible effects on the fight against crime and terrorist acts. These effects fall into two main points: the first is deterrent crime prevention, and the second is the collection of security information and the production of intelligence.

V.3.2. The direct effect

Surveillance systems can fight the crime and the terrorist acts directly in three stages:

1. Before the event: suspected incidents or events can be detected, by observers during interactive monitoring or by video analytics, and can be prevented. This is called proactive prevention. This is the case, for example, when an ANPR system detects and triggers the alert when a wanted car pass by a gantry, or when a face recognition system alerts the observer about a suspected person passing in the view field of a camera, etc. 2. During the event: events are followed by observers, video analytics, as well as live incidents reported by citizen to police emergency rooms. Here, the problem is to locate quickly the incident, help the police to assess the situation and appropriately intervene, and prevent the situation from escalating. Video analytics can help the operator to quickly locate the incident by searching some features, tracking cars or persons, counting people in the crowds, collecting information about involved people (faces, descriptions…) and cars (models, plate number, routes…). 3. After the event: it is possible to return to previous events with the aim of investigating an incident by the relevant police units and arresting and convicting the criminals. Video analytics can help investigating the archive with appropriate and advanced features, which save dramatically the time wasted to find the incident, and recognize the involved people and cars and their habits, trajectories and behaviour patterns, hot spots and other critical information.

In all the above points, direct or indirect, surveillance systems can be a game changer in public security and other fields when using appropriate intelligent video analytics. But the main problem remains where the end users do not get what they need. In the next section, we focus on this point and we propose some practical needs and improvements to video analytics. NB: It is worth to mention here that, many contradictory opinions exists about the use of the surveillance systems and its risk on privacy and liberty, where the fear from the "Big Brother" is increasing. In plus, many claims about the non-effectiveness of using surveillance systems remain as it is only displacing the crimes, adding to that, the allegations about the surveillance systems as not cost effective. These two mains points are the subject of many researches in this field; here it is not the right place to amplify the explanation about it. Shortly, after doing many researches about these two important points, and after practicing and manging such a system during three years, one cannot deny that, such a system can be limitary miss-used in the wrong hands. Hence, the firm regulations and procedures, to not allowing that from happening, take role; in the other hand, the installation of a surveillance system should not exceed the public places. Concerning the cost effectiveness point, one of the main strategies of fighting crimes is to displace them; also, there is a miss-conception, when measuring the crimes statistics before and after the installation of surveillance systems, which is not taking into consideration that, after installation of surveillance cameras, the detection of the crimes increase yet not the number of crimes. Moreover, one question is placed at the disposal of the reader judgement: if a surveillance system helped saving only one person life, and sure it can help many, how one can estimates the cost of this person's life?

V.4. Filling the gap between practice and theory

There is a big gap between the research field and the application domain. What is considered as done in the research field may be considered inefficient or insufficient for the end user. Let us take, as a quick example, the simple video motion detection which has been used to provide a way for alerting the operators or activating the recording systems. While this kind of tool is widely installed in commercially available surveillance systems, motion detection still has many drawbacks, as it can be falsely triggered by non-motion incidents such as those due to variations in the illumination or weather-related changes. These false alarms are a significant disturb for a police officer, costing him to be alerted and to prepare for an action, which is thus time and effort consuming.

While in the research field, video analytic studies address very advanced subjects, on the other hand, in the market, the production is sometimes still not at the needed level. This discrepancy could be explained as kind of miss understanding of the real needs to improve the use of the system, from three perspectives: -From the end user point of view: the system managers or CCTV control room managers (for example the police) may not know how to express their needs in terms of video analytics and may have difficulties to estimate a priori its power and impact. -The companies and industries are the link between the end users and research laboratories, and so, they focus on valuable technology transfer leaving on the side tools which may generate a too low profit, even if these ones may be of interest in some specific cases. -As a result, the research field, not always take into accounts the real needs, or real feedback, while it is spreading its efforts in many directions.

We should note here, that many efforts and conferences have been held trying to raise the awareness about theses gaps and to approach these mentioned fields perspectives (ICDP conf, 2017), [START_REF] Zaman | Advances in Visual Informatics[END_REF]). Yet many works still have to be done.

According to the research done, and my experience as manager of a new CCTV system, some difficulties on both sides, in the research field and in the practice field, must be highlighted:

V.4.1. Main difficulties and propositions in the research field

-Need for strategic planning for problem solving One important difficulty every researcher suffers from is not knowing where and what to do in an already known field, especially if there are no normalized procedures, datasets, and results to refer to. Even if it's the researcher role to find where he or she is standing among the studies, but without a strategic planning, this could become quickly a waste of energy, talent and time. Having such a plan, the researchers will not focus their efforts on an already existing field with previously good results, conversely the dedication would be to start a new one or at least to continue where the previous finished.

-The non-availability of pretended good algorithms One of the most frustrating cases occurs when reading about an algorithm producing good results in some field, without having the capability, in most of the cases, to use it as it is without major efforts, because of the non-availability of a piece of runtime software or because its high cost, or if appears after testing the algorithm that the claims about the good results are overestimated. In the opposite case, having an algorithm with very good results, and available to the public, it could be a great contribution in the scientific field, for example, a tracking and segmentation algorithm.

-The great need of real datasets Working on video analytics, especially with AI oriented algorithms, raises a great need of huge databases of real video surveillance. Even though there is an access for a one, another need pops up to front, corresponding to the lack of dataset annotation and the lack of positive cases of significant incidents. With respect to the applicable regulations and procedures, the solution would be the continued cooperation between the laboratories and surveillance systems owners, and to take advantage from all marked incident scenes and annotation taken from their side.

-Tracking and segmentation problems In the video analytics field, even with the existence of more sophisticated methods and algorithm, there is still a great need to optimize indispensable objects segmentations and tracking algorithms, because it is the base of most of the objects features and the first step of many systems. Most of the algorithms drawbacks and inaccuracy are mainly due to two problems:

1) The shadow and illumination: it is hard to segment the object from its shadow especially when having a difficult background and changing illumination. Also, it makes it difficult to segment the sub-object parts.

2) The occlusion problem: we may consider two types of occlusions, one with a background object (and this is the easiest one), and the second with another moving object, where the occlusion may take long time when there is an interaction. In that case, the system loses the objects during the interaction and may identify it after as being another object, which leads to more post-processing steps (recognition and reassignment).

V.4.2. Main difficulties and propositions in practice field

As already mentioned, there are two main problems facing the surveillance control room management, the shortage of active monitoring following the need of automatic alerts, and the difficulties investigating the archives:

-The shortage of active monitoring Actively monitoring a whole city with thousands of cameras is impossible, even when having a big number of operators. Moreover, even with low number of observed cameras, most of the incidents are logically miss-detected in like-wise manual system due to natural limitations from deploying indispensable solely human operators facing CCTV screens. These miss-detections could be caused by:  The limitation of human being capability: the human attention span is limited and tasks that require intensive sustained vigilance such as monitoring CCTV feeds should be covered in brief shifts of 20 minutes and maximum 30 minutes before resting for a while, and covering a limited number of cameras. Deficiency in human resources makes these conditions hard to fulfil. Therefore, long hours watching an excessive number of video screens is the real daily bases of many surveillance systems.  Boredom from monitoring is a real issue also, nobody take it into count.  Lack of a priori and readily accessible knowledge for what to look for  Distraction and interruptions by additional responsibilities such as other administrative tasks [START_REF] Gill | Control room operation: findings from control room observations[END_REF].

Considering the above human limitations, an effective live monitoring therefore requires a large number of operators that will inevitably increases the cost. Confronted to this reality, plus the limited resources, the management tend usually to allocate little resources to the live monitoring and prefers to focus on the passive one. This choice tend to be the right one considering the small number of incidents that can be detected with human live monitoring compared to the other duties that could yield a better return. However, considering the great importance of live monitoring, using video analytics that can actively alert the operators on possible incidents becomes the best support to the human monitoring.

-The difficulties investigating the archives Searching hundreds of hours of video footages for an incident without having exact location and time is barely impossible. Especially when we are not sure if the incident will shows up inside the Field of View (FOV) or not. Sometimes, it is only some meters farer or some minutes later than the searched incident occurs. Most of the times, the incidents are not in the FOV, but we still need to search the surrounding areas for a possible passage of a motorcycle or a car or a person with some specific features. All the activities mentioned above are extremely time-consuming, and they are done manually.

Having video analytics than can decrease dramatically the search time, by using some features, will be a great improvement. Despite much advancement in the field of automated surveillance, video analytics is still facing some challenges when it comes to real world conditions.

In the following section, we list some of the main problems and propose some improvements, on the level of systems and software, which could be very beneficial for the end users:

V.4.2.A. System improvements propositions

-Integration with other systems A Video Management Software (VMS) is not effective alone. Surveillance systems using a VMS would be more effective when it is integrating with other systems. Let us mention for example: 1) VMS integrated with ANPR systems (including database of wanted cars), where the wanted (marked) car in the ANPR system, when triggered, can be highlighted by the VMS cameras. 2) VMS integrated with face recognition systems (including database of wanted people), where the fixed and PTZ cameras can search for faces to be compared, and then highlight them if they appear as wanted persons. 3) VMS integrated with a mapping system: surprisingly most of the VMS come without a mapping system, which is one of the most important and needed systems for crimes analysis or traffic management. A mapping system can handle all the geographic and context-based database of the area under surveillance, and by this mean the abstract description can be improved by combining the contextual information (place, location, popularity, weather, temperature, lighting …) to generate more semantic and meaningful descriptions. 4) VMS integrated with the "Computer Aided Dispatch" (CAD) systems: where it can show the live location of moving cars (for example patrols). 5) VMS integrated with "Internet of Things" (IOT) systems: where IOT allows to aggregate data from various sensors, which is a very vast domain. 6) VMS integrated with Business Intelligence (BI) tool: where hot spots of all the incidents can be highlighted according to live analysis of these incidents, helping the operators to focus on the most possible places where and when incidents may occur.

All the above mentioned systems can be handled by an appropriate mapping system where all pieces of information can be seen as embedded layers.

-The VMS should be more than management The VMS it is not only for the video management, but it can be considered and improved to be a collector of data, connector of devices and provider of analysis, among others.

As it has been mentioned before, all the collected data from video analytics and descriptors will accumulate information to form big data that can be learn through clustering, and regression for knowledge modelling and event prediction. Hence, this should lead to new opportunities and new challenges. Data mining and AI can induct prediction and proactivity. For example, the system can latter reference the relation between a specific person or object or car and a specific location, and indicates its hot zones. Another example is when a target enters a scene, the learnt routes provide typical patterns of behaviour, and probabilities are assigned about possible exit points. Later, atypical behaviour can be identified with targets that do not use the established routes and will alarm the security personnel.

In terms of architecture and design, in order to reduce the excessive bandwidth needs for transmission and the computational load of the central processors, there is a trend in video analytics practice to prefer distributed intelligence solutions, which consist in locating more video processing or intelligence analytics at the level of cameras (or sensor).

Nowadays, with the great progress we witness in the domain of AI and DNN, in my opinion, the next-generation of intelligent video surveillance analytics systems will induct DNNs and more generally AI as far as we can identify that there is a great need.

The importance of user-centred design or user-centric models is now widely recognised in video indexing and retrieval. But we should not forget the excessive bandwidth and the computational load. Both types (centralized and distributed) of systems should coexist. Hence, those systems could take benefits from the advantages of both designs. Keeping some level of distribution, it can furnish more natural and flexible user interactions. Relevant feedback from the user is essential so that the system can perform better; this can optimize the learning of the system and its performance.

Especially when, the limited amount of positive cases to train recognition algorithm adds difficulties in detecting these so-called rare events.

When seeking for performance, there is no escape from coexistence of both systems, where the system should be learned locally, at level of each camera alone, and at a global level, fed with all outputs of cameras, to "see the whole picture (puzzle)" of the current state of an area under surveillance.

The concept of learning from the data is widely appropriate for video surveillance, since it is possible to allow the system to learn by observing scenes activity over long periods of time, where the scene context and structure influences, directly or indirectly, the way that objects act. Therefore, specific type of events or incidents may be associated with specific regions. For instance, doors or gates oblige people to pass by; roads constrain vehicles to move along in a particular direction, etc.

This strengthen our point that what is needed, is merging heuristic-based and learning approaches, where we can learn dedicated heuristic features for better understanding the area, the objects, and the incidents, seeking better performance and generic abstract description. Accordingly, each area seen by a camera has its own particularity (routes used, speed level, density…), added to the whole city common characteristics (weather, demographic, geo-economic…).

-Real timeliness Useful video analytics algorithms for surveillance systems should be real time. However, very accurate and robust real time analysis needs intensive computing processing. And intensive computing processing needs lots of investment. Again, one possible solution would to make the accuracy easily adjustable by the users so it could match their computing capability when doing real time surveillance. Besides, this could also open new market for the software companies. Market that are currently inaccessible because of the investment barrier.

-Standardization for video analytics Since there are so many hardware types and video file format, the integrated systems with high diversity are facing hard time to exchange data and to view or access the video contents. This case will be more critical for video analytics, as high-level semantic descriptors are required to represent properties of objects, events, scene contents, and so forth.

V.4.2.B. Software improvements propositions

-False alerts or false negative In the research field, having a precision rate of 85% and above can be considered as good results for some tasks. However, in the practice, the reliability of intelligent video analytics is a paramount issue, since 1% of error can be catastrophic. For example, a false alert of 1% for 1 million cars entering Beirut city daily, means 10 000 records that need to be checked manually. Additionally, frequent false alarms induce mistrust in the operators, who quickly tend to ignore the system. Therefore, improving accuracy even by some percentage could have tremendous positive impact on the work of the end users [START_REF] Velastin | CCTV Video Analytics: Recent Advances and Limitations[END_REF].

-Miss-detections or False Positive CCTV control room managers usually mistake miss-detection for false detection. The miss-detection is a very critical problem and most video analytics companies don't communicate very openly about it. For example, the ANPR, despite of being one of the most reliable systems, it still has its miss-detections rate even if it's very low. And Failure to detect, for example, a suspected or a stolen car going into the country to be used in a bombing, is not an acceptable error. Again, increasing precision in general will help. But for this particular problem, one possible solution is to make threshold that triggers the alert of the algorithm adjustable, in a simple way, by the end user. This way, the operator can, for critical cases like an imminent threat for example, lower the threshold (even if this might increase the number of false alerts). Then it will be up to the end user to find the right balance between the miss detection and false detection.

-Query parameters Due to the rapid increase of the number of cameras used in video surveillance and the huge amount of footage that can be produced, the challenge of video analytics is to extract meaningful information in order to produce high-level semantic resources. These resources can be later used to search for a specific content. Being able to query the video database for using combined parameters could help dramatically the operators looking for an incident and save a lot of time. The existing products on the market allow searches according to some important parameters, like the colour, direction and type of an object. But in practice, there is a wider range of needs, like the ones provided by this thesis. It would be good if the wide range of features involved in the algorithm can be taken as parameters for searching. This set can be enlarged to some recognition-based parameters. Let us mention:  Object type, like animals, bicycle, truck, motorcycle, etc.  Same object like the same person, car, scooter, text and letters, etc.  Shapes like box, logo, tattoos, wheel rim, hat, knife, umbrella, car lamps, etc.  Special marks, like car with damage in a special area, or broken light, etc.  Night special marks, like, coloured lamp light (blue, neon…), lamp shape, the distance of car lamps, flashers…  Many other parameters may be used also.

It could be also very useful to have the ability to simulate the scene by designing a scenario, for the system to search for similar ones, including all possible parameters like objects, movements, and interactions. For example, a person with a box comes from a specific route… Moreover, it is very beneficial if the search results came with percentage of reliability, indicating the percentage of matching the searched criteria. NB: many of the above-mentioned parameters and features could be valuable also to be used for alerts.

-Tracking and recognition

The system should be able to recognize and track, through cameras, even far ones: 1) Wanted persons.

2) Wanted cars.

3) Any selected object. An idea is to assign high probability to the surrounding cameras to search in it. Also, when marking and selecting a car or person, it would be beneficial if the movable (PTZ) cameras can automatically seek for more description of the tracked object (car plate number, face…).

-Detection on some special cases It would be very helpful if the system can detect also the "non-existence" of a feature, for example:  For cars: detecting cars without plate numbers.  For motorcycles: detecting motorcycles without plate numbers and persons on motorcycles without casks  For people: person "without face" (whom faces are covered).

-Behaviour and body language analysis Detecting special behaviours and analysing body language locations could help preventing crimes. For instance, detecting the special behaviour of the thieves (face down, waiting, surfing …) could help prevent a robbery.

-Improve robustness A major challenge for real world scenes is the dynamic nature of real world conditions. Achieving robust algorithms is a challenge especially under illumination variation, weather conditions, under view changes, existence of multiple objects, occlusion, deformation, shadow, reflections, video noise, and moving background. Significant research and advances in solving these difficulties have been achieved, but user can profit still from lots of improvement.

-The night time surveillance

The night time is one of the biggest problems which CCTV observers and investigators suffer from. Even when using Infra-Red cameras, a wanted car or person may be detected but no more details or description will be available. Moreover, during the sunset and the sunrise even the IR is not effective. What is needed is to lighten all the surveillance area with white lights, as it is better than yellow. Any enhancement that can be done at the software level, it would be of great benefit.

-The rain This is a common problem for all surveillance systems, when not only the water sticks on the lens but also dust, or more generally when the rain makes the vision impossible. For the water to slide down from the lens, some new lenses were introduced. Also, some systems use a special solution to spray the lenses. But the problem of the vision remains, and need more attention, maybe some kind of filtering at the level of software may improve the vision.

V.5. Conclusion

Visual surveillance systems supported by appropriate intelligent video analytics can be an effective weapon in the hands of the law enforcement agencies. In this chapter we overviewed the surveillance systems and their usage for fighting crimes. We highlighted the existing gaps between the research field, production companies and end users of surveillance systems. Many propositions, inspired by practical and theoretical experience in this field, were made to narrow this gap. These propositions encounter the research field, and the practical one, especially at the level of future intelligence video analytics development and integration with other systems. Some of these propositions are innovative yet simple to be applied, which can bring great benefits and optimize the use for surveillance systems operators for live monitoring, investigating, and analysing the crimes.

VI. General Conclusion

In this chapter, we first list our key contributions in the field of video surveillance analysis and description. Then, we will show how the questions mentioned in the introduction find an answer thanks to our VSSD approach. Finally, we indicate interesting directions for future research in this field.

VI.1. Key contributions

The key contributions of this thesis are the following ones:

-Our VSSD approach introduces a new heuristic way to discriminate between deformable and non-deformable objects in the scenes. -Our VSSD approach presents a new generic flexible and extensible ontology for video surveillance scenes description. -Our VSSD approach introduces new concepts concerning mediation and interaction at a distance, deformable and non-deformable objects, abstraction in description, and a new manner of categorizing the scenes. -Our VSSD approach implements the original new classification idea of distant vs physical interaction, while other works focus only on the physical one. Moreover, detecting distant aggressive interaction can alert the surveillance control rooms' observers at early stages, giving them precious time to act. -Our VSSD approach presents a set of new features dedicated to the interaction classification process. While many methods focus on exporting features using convolutional networks from the frames, in VSSD approach we took into consideration the distinctive propriety of videos which is the temporal relation through these frames, and we focused on selecting valuable features which can influence or get influenced by the interaction, like the object direction, shape, deformability, Hu moments, speed, etc. These video scenes important features can be used for generating alerts and intelligently investigating the archives from real case perspective. -Our VSSD approach integrates the traditional methods of the features extraction along with machine learning and DNN of the interaction classifications. This integration allows our approach to benefit from the advantages of both methods, by giving more control when selecting the features and more results' accuracy when classifying. -Our VSSD approach provides a novel direction to work in generic abstract domain. As mentioned, working with video surveillance is a complex problem due to the scenes diversity, like scene location, environment, object types, actions and interactions. To simplify the problems, some research added more assumptions, and this may have improved the results significantly but it limited and restricted its applicability in real world. To encounter this, our VSSD approach is kept generic, not restricted to any of the scene categories, and abstract, not semantic. -VSSD can be easily extended and improved without major changes in the overall process. -Our VSSD approach presents a new ontology-based and non-contextual way for video scene description using well-structured generic templates. The new structured templates contain the main information reported by the police in real case incident descriptions. Consequently, the textual output description can be generated automatically as draft reports to be based on. -Our VSSD approach allows the end user to tune the verbosity of many key description characteristics. -Finally in chapter V, we developed several propositions on both sides, in the research field and in the practical field, driven by practical experience to reduce the existing gaps between the surveillance systems operators' needs from one side, the research field and the commercial (industry) field from the other side.

VI.2. In Summary

As a summary, we propose here to go back to the questions identified in the introduction of this thesis, and to answer them with collected elements taken from the previous chapters: a) How should the video description system be built in order to be more efficient and useful for different surveillance systems?

In general, surveillance systems differ from a system to another mainly by the diversity of the scenes.

To build an efficient and useful description system that can suit different surveillance systems, the diversity of the scene types, the environment, the objects types, the actions, and the interactions should be taken into consideration. In our ontology-based approach, as seen in chapter II, we kept our approach generic, not restricted to any of the scene categories, and abstract, not semantic. Also, as seen in chapter IV section IV.3, our system preserves the modularity where each of the composing algorithms can be treated as independent module. The algorithm can be replaced or altered by adding more information as input or producing more outputs like features. We want to use our system for intelligent search, generating alerts and extracting textual reports, also our system should be able to describe any objects abnormality and the interaction between two of them. For all of that, as seen in chapter IV section IV.3, our approach concentrated on extracting and analysing simple yet influencing visual information in videos surveillance which can feed these targets, we mention the object direction, shape, deformability, size, Hu moments, speed, position, trajectory, existing of interaction, interaction type, and interaction aggressiveness, etc. These characteristics are very useful for Beirut CCTV control room as search, alerts and reports, see chapter IV and Appendix VIII.8. Nevertheless, our system is not restricted to these characteristics where others can be added to the mentioned functionalities by simply adding these extra characteristics, from the 2305 features, to the scene activity characteristics matrix and the description.

c) What should a system describe? How to decide when generating a textual description along the time dimension? How much the description is practical and responds to the user needs?

The description produced by a system varies from a surveillance system to another according to the needs and goals of the corresponding control room management. It could be a full description of the footage (having, at each moment, a full description of all the features of the existing objects) or more simple description for only certain events or behaviours (E.g.: abnormal activities like law violations, and others). This situation may be highlighted by a change in speed, or shape, or trajectory… As seen in chapter IV sub-section IV.3.8, our approach is able to produce three types of description: the full description, the short description and object life cycle description.

For appropriate description, we presented a simple threshold-based method to detect all features' abnormalities.

As the system will be used by the observers, investigators and analysts, it should respond to their needs from one side, and from other side it should not submerge them with useless outputs. In our approach, for more practicality, the verbosity of the system outputs is adaptable and controlled by the end user (by selecting what features must be used as triggers for detecting abnormality and generating description, and by controlling the density of generated descriptions).

d) How to design a powerful sentence generation model? What an adequate textual/sentence representation contains? What is the best combination of different components of a representative sentence? A sentence model for a video surveillance system is powerful when this output model can help to generate the real case incident reports done by the investigators, and interpreted by the analysts.

In real cases, as mentioned in chapter IV, when an incident occurs, five main points are mainly needed for a scene description, also known as the five Ws (Who, what, where, when and why (which is often replaced by how)).

As seen in chapter IV section IV.5, the video description is subjective and uncertain and there is no best combination of the representative sentence, only an adequate and preferable one. In our approach, we embedded all the mentioned pieces of information in the sentence description. Also, we used suitable structured templates; similar to the ones used for police reports, reflecting our reports needs as experts in the practical field.

VI.3. Conclusion and Perspectives

If properly implemented and used, visual surveillance systems, supported by intelligent video analytics, can become a very effective weapon in the hands of the law enforcement agencies. It can not only help arresting and convicting criminals in very fast and efficient way, but also (and most importantly) it can also prevent some crimes from happening.

This thesis looks fundamentally at the problem of describing important contents in videos surveillance scenes, based on a new generic context-free ontology, focusing on objects interactions. While analysing and understanding a wide variety of video scenes, our approach introduces new concepts and highlights important features for better classifications of video object interactions. These features, used as key parameters in video analytics tools, are much suitable for supporting surveillance systems operators by alerts and intelligent search. Moreover, our system outputs can support investigation reports, according to investigators needs, with many types of automatic textual descriptions based on new well-structured generic rule-based schemas or templates.

In this thesis, we did not pretend to build the ultimate highly intelligent surveillance analysis and description system. This research can be seen as a step forward toward this target. It can be taken as a set of propositions that could improve the existing video surveillance systems in many ways.

Many imperative works can be done in the future succeeding this thesis. This work can be extended by fulfilling the remaining concepts of the proposed ontology. As in this work, the objects categorization was sufficient as deformable and non-deformable, also the sub-objects and the interaction subtypes are not deeply investigated, so one can start to analyse the deeper level of these classifications. For objects categories a lower level of deformable and non-deformable object classifications can be reached, namely humans, animals, plants, machines and inert objects. Concerning sub-objects, the deformable object articulations and its movements can be the focus of many studies, especially its relation and correlation with the object interaction. Also, interactions subtypes can be investigated to add more layers to the classified ones (distant vs physical, and aggressive vs peaceful), for example the interaction has bad influence or good influence on an object, etc.

In plus, for having better results in tracking and segmentation which can have a huge positive influence on the system results, the current tracking and segmentation algorithm can be replaced by a more recent algorithm based on deep learning, potential YOLO v3 or Mask R-CNN.

Additionally, as applying this description approach on other scenes' types (according to our types seen in chapter IV) seems simple, an interesting future work will be to apply it on more complex scenes, showing interactions between more than two objects.

Moreover, the need for more data for learning the classifiers is still a big issue; especially real databases, diversified annotated, and classified. As the head of Beirut CCTV control room, I have the chance to manage the control room's databases and its annotations, about real incidents and events, in a suitable way for later tests, so I will take advantage of this rear opportunity to drive these tests and researches. This can be led to more improvement in both fields, research as automatic video analysis and practical as law enforcement applications. Furthermore, to optimize the automation process of the entire approach, as well as to generate specific and practical scenes descriptions, integrating contextual information (mentioned in the chapter II) would make the description more relevant to the operators. As an example, areas and spots of the camera field can be named into more meaningful and contextual descriptions like, in front of the "name" market or intersection of "name 1" road with "name 2" road. A practical method to introduce this information is to combine the Video Management System (VMS) with a mapping system, as mentioned in chapter V. No training is mandatory to reach that level of description.

On top of that, as working on thousands of hours of videos surveillance footage, all the output data from our system, when applying, can form a big data. This big data or metadata collected and accumulated over a set of few months can be then learned through clustering, and regression for modelling and prediction of the objects behaviours and interactions.

From another perspective, driven by our personal experience as researcher and surveillance system manager, in order to encourage more cooperation, we highlighted the existing gaps between the research field, the video analytics companies and the needs of surveillance system end users. On a large scale, several possible future directions in the practical field and the research one were extensively discussed, keeping in mind that the surveillance systems are only tools that cannot operate alone and that should be integrated with other systems, within a larger public safety and security strategy. From these propositions, we mention the one about answering the need for flexibility when it comes to adjusting the algorithm thresholds allowing the end user to have more control based on his or her needs.

With the current advancement in the field of learning methods, many researches of different domains tend to shift their approaches to rely completely on learning strategies at all levels. We believe that it still early in the domain of video surveillance to rely completely on learning strategies. Nerveless in the future, new technologies and software supported by AI and more specially DNNs will change the face of surveillance systems. Until then, we believe that, for better intelligent video analytics products encountering all the needs of surveillance system management goals (live monitoring and crime solving by searching the archives), a hybrid combination of learning and traditional video understanding approaches shall still be considered. The proposed VSSD can be seen as an example of this combination. On the other hand, these coming changes need to be matched by the readiness acceptation of human operators and managers to meet the challenges ahead and to make the appropriate changes in security policy and planning. The users of these systems, from law enforcement agencies to criminal justice systems, will need to evolve and to adapt quickly to these new technical changes and opportunities.

VIII.2. Appendix 2: Motion estimation techniques

We should note a duality between motion detection and estimation where the motion detection can be used to find the active region of motion in frames. The inactive region is assumed with zero motion vectors. Then the motion estimation can be applied to active regions only. The early detection of regions with zero motion vector leads to significant reduces in computation.

Also another duality should be noticed, between motion estimation and segmentation operations. In order to correctly estimate the motion, regions of homogeneous ones need to be known. Contrariwise, for better segmentation of these regions, it is essential to apply, previously, motion estimation. This problem can be tackled by joint motion estimation and segmentation techniques [START_REF] Pesquet-Popescu | Motion Estimation Techniques[END_REF].

Another perspective to look at those two approaches (motion detection and motion estimation) is to see them as same approach. To estimate a motion vector, that means detect it and estimate it resulting into foreground segmentation of moving objects, that's why approximately all the motion estimation methods can be and are used for motion detection. But concerning the detection of the motion, can it estimate the motion! Yes, instead of assigning values to each pixel in the visual input, which can be a complex task, detecting motion focus on extracting of moving features (here the feature is a moving objects in the scene), then estimate its position in the next frame, revealing the vector motion of the object. This approach lay under the semantic level (object-based level), where can be called object-based detection or Object-matching.

Many good surveys mentioned this with details (Vishwakarma & Agrawal, 2013), [START_REF] Candamo | Understanding Transit Scenes: A Survey on Human Behavior-Recognition Algorithms[END_REF][START_REF] Hu | A Survey on Visual Surveillance of Object Motion and Behaviors[END_REF], [START_REF] Oliver | A Bayesian computer vision system for modeling human interactions[END_REF].

The most used methods for motion estimation are:

1. Background subtraction: Background subtraction is one of most popular motion detection methods, especially when working with a static camera. It detects moving regions in an image by calculating the difference, pixel-wise, between the reference background image and the current image. Pixels having the result of difference near null are assumed as background pixels, and the rest of pixels belong to a moving object. These methods assume that background changes are much weaker compared to object changes. Background image is simple to use, but they suffer from shadows and reflections of moving objects which may be highlighted in the difference image. In addition, it is extreme sensitivity to changes in luminosity or dynamic scenes. Therefore, to reduce the influence of some of these changes, it is highly recommended having good model for background, [START_REF] Gandhamal | Evaluation of background subtraction algorithms for object extraction[END_REF], [START_REF] Haritaoglu | W4: Real-time surveillance of people and their activities[END_REF], [START_REF] Mckenna | Tracking Groups of People[END_REF], (Stauffer & Grimson, 1999). 2. Temporal differencing: it performs pixel-wise differences between consecutive frames to extract moving regions. After that, to determine changes, a threshold-based function is used. The extracted moving sections are, then by applying a connected component analysis, clustered into motion regions. This approach is adaptive to dynamic environments, but sometimes fails to extract all the relevant pixels. Other problem is that non-moving parts of objects are non-detected. Besides, that these kinds of methods are very sensitive to noise and luminosity. Furthermore, temporal changes between consecutive images may be detected in areas that are close to object boundaries and in uncovered background. In addition, deposited or removed objects cannot be correctly detected using successive images [START_REF] Verma | Analysis of moving object detection and tracking in video surveillance system[END_REF], [START_REF] Aishy | Object and Event Extraction for Video Processing and Representation in On-Line Video Applications[END_REF], (A. J. [START_REF] Lipton | Moving target classification and tracking from real-time video[END_REF].

Optical flow:

It is defined as the apparent motion of the brightness pattern. In other terms, it captures the spatial and temporal pixel intensities variation in image sequences. It is similar of velocity measurement, and reflects the image variations during a time interval due to motion. Optical flow is used without any prior knowledge of the frame content, and is suitable for large variety of motions. Also, can be used to detect a moving object in the occurrence of camera motion. Optical flow can be calculated with multiples methods (Differential techniques (Variational techniques), Region-Based Matching (Correlation-Based Methods), Energy-Based Methods (frequency-based methods), Phase-Based Techniques), described in Appendix 1.The optical flow analysis approach gives very dense and approximately accurate results, in each pixel. It is considered to be one of the most detailed and rich motion representations of an image from a video signal. In general, this method can be applied if the intervals between consecutive images are very short, and if no significant change occurs. On the other hand, flow computation methods are computationally complex and time consuming methods. Besides, shadows and reflections of moving objects can be highlighted in the result image and non-moving parts of objects are non-detected. [START_REF] Barron | Performance of optical flow techniques[END_REF], (D. [START_REF] Meyer | Model based extraction of articulated objects in image sequences for gait analysis[END_REF], [START_REF] Meyer | Gait Classification with HMMs for Trajectories of Body Parts Extracted by Mixture Densities[END_REF].

The Block Matching Algorithm:

Block matching is a special case of the region-based approach. In this approach, motion estimation algorithms are based on the matching of blocks between two frames, with the objective to minimize a dissimilarity measure. For that, current frame is partitioned into blocks for purpose to find out the corresponding motion vector for each block according to its relative displacement from the previous frame. The same displacement vector is assigned to all pixels within a block. Block matching is widely used in video coding for transmission or compression purposes.

Another important reason for this wide use is the low computational cost it involves.

According to [START_REF] Love | An Empirical Study of Block Matching Techniques for the Detection of Moving Objects[END_REF] Concerning Matching criteria, also known as error or matching functions, we can mention:

-The sum of the absolute values of the differences in the two blocks (SAD).

-The mean squared error (MSE).

-The sum of squared errors (SSE).

-The sum of absolute transformed differences (SATD).

-The mean of the absolute values of the differences in the two blocks (MAD).

-The mean of the square of the differences in the two blocks (MSD).

-The sum of the non-matching pixels in the two blocks, where a match is defined as the difference absolute value is less than a threshold (MPC).

Some Block matching approach can give a very dense motion field (each pixel), even more, some others can give sub-pixel accuracy, but the computational cost, in either case, is extremely high. Using the Full Search Algorithm (FSA) can give very good results but it is an extremely time consuming method. Besides, shadows and reflections of moving objects can be highlighted in the result image and non-moving parts of objects are non-detected. To know more about Block matching Algorithm the reader is referred to [START_REF] Khammar | Evaluation of different block matching algorithms to motion estimation[END_REF], [START_REF] Love | An Empirical Study of Block Matching Techniques for the Detection of Moving Objects[END_REF], [START_REF] Patel | Review and comparative study of motion estimation techniques to reduce complexity in video compression[END_REF].

5.

The feature-based approach: also known as feature correspondence or matching. This approach is based, as first stage, on extracting a set of relatively sparse and discriminatory features in the corresponding images, such as edges, corners or other distinguished points. Inter-frame correspondence (matching) is then established between these features, as second stage, to remove matches that do not correspond to the actual motion and give a set of motion vectors. The resulting motion field is estimated at those feature points only. The uncertainty in determining invariant, accurate and reliable features due to various image distortions at the feature detection stage. At the feature matching stage, the well-known correspondence problem of ambiguous potential matches occurs. Main works were carried out especially on feature detectors and recently good results were achieved; for that reason the most recent SFM approaches prefers using the feature-based approach… Motion estimation based on correspondence of interest points (feature points) works well for inter-frame non-small time intervals. Sparse and non-regular correspondence points are also detected. Highly textured objects which are common in real scene cannot adequately be handled by primitives like edges, lines and corners… Besides, shadows and reflections of moving objects can be highlighted in the difference image.

For more information for the feature-based approach, the reader is referred to [START_REF] Heel | Direct Estimation of Structure and Motion from Multiple Frames[END_REF], (Farin & With, 2005), [START_REF] Szeliski | Feature detection and matching[END_REF], [START_REF] Szeliski | Feature detection and matching[END_REF], (Farin Dirk, de With Peter H.N., 2005), [START_REF] Torr | Feature Based Methods for Structure and Motion Estimation[END_REF].

Next, Table VIII-1 compares major approaches and classifies them into: Poor, Moderate, Good, and Very Good.

NB: The assessment of those approaches is aimed for the approach in general. Bad assessment means that the approach in general suffers from this kind of problem, but does not deny the existence of some methods able to overcome the mentioned problem.

For more information the reader is referred to [START_REF] Aggarwal | On the computation of motion from sequences of images-A review[END_REF], [START_REF] Pesquet-Popescu | Motion Estimation Techniques[END_REF], [START_REF] Dufaux | Motion estimation techniques for digital TV: A review and a new contribution[END_REF], [START_REF] Patel | Review and comparative study of motion estimation techniques to reduce complexity in video compression[END_REF], (Patel M. B., Kshirsagar R. V., Nitnaware V., 2007). It appears obviously that, in the graph, a big plateau of probability = 1 exists, and that could not be correct. For example for N = 40 and N 2 = 1, 2 or3…22, the probability = 1 and that it totally wrong.

If we compare those results with the numerical and graphical results of the maximization expressions: arg max 𝑁,𝑁 2 𝑃[(𝑋 ≥ 𝑁 2 ) ∩ (𝑌 < 𝑁 2 )], we find that the graph of the maximization expression (eq. III-15) is descendent on the right side when N is growing and N 2 is remain respectively small. And this it seems more logical. For example: if N = 40 and N 2 = 3, the probability = 0.019.

Finally, the maximization expression must contain both terms X and Y, where we have to validate all the correct classification X, and reject all the false classification Y. In this scene of moving person on the "Stairwell", we can see a series of nondeformable movements, when only the upper body is shown and moving in a quite translating way like the motion between frame 28 and frame 31 (motion frame 31); But after that, legs appears at frame 36, and motions became clearly deformable like the motion frame 44. All of those motions classifications can be clearly inferred when comparing percentages of correctly mapped points to the corresponding threshold (see tables above). Plus if we put the percentages of correctly mapped points of those frames scene in a graph, it appears clearly how and when the motion became deformable: In this scene of a running person, for this motion of frame 28, we can see clearly that the percentages of correctly mapped points are smaller than its corresponding thresholds whatever the normalization for F or H, and then this motion is deformable. Almost all motions in this series are deformable motions.

NB:

For us, the decision will be taken according to the Percentage of correctly mapped points when Normalization = 2, 𝜸 𝑭 𝟐 =2.  Scene of "Highway 2": o Frame 97:

In this scene of a car on the highway, for this motion of frame 97, we can see clearly that the percentages of correctly mapped points are bigger than its corresponding thresholds whatever the normalization for F or H, and then this motion is non-deformable. Almost all motions in this series are non-deformable motions. NB: In our experiments, the decision will be taken according to the Percentage of correctly mapped points when Normalization = 2, 𝜸 𝑭 𝟐 =2.2 and 𝜹 𝑭 𝟐 = 80.16. It's clear that we can infer the non-deformability of motion from the Homography too. In this scene of a person planting a bomb, for this motion of frame 117, the person is moving in a non-deformable way, we can see clearly that the percentages of correctly mapped points are bigger than its corresponding thresholds whatever the normalization for F or H, and then this motion is non-deformable. Almost all motions in this series are deformable motions. NB: In our experiments, the decision will be taken according to the Percentage of correctly mapped points when Normalization = 2, 𝜸 𝑭 𝟐 =2.2 and 𝜹 𝑭 𝟐 = 80.16. It's clear that we can infer the non-deformability of motion from the Homography too. Another interesting works can be seen for object detection in video [START_REF] Vorobjov | An Effective Object Detection Algorithm for High Resolution Video by Using Convolutional Neural Network[END_REF], [START_REF] Pathak | Assessment of Object Detection Using Deep Convolutional Neural Networks[END_REF], and for detection and classification in video, Focus [START_REF] Hsieh | Focus: Querying Large Video Datasets with Low Latency and Low Cost[END_REF].

VIII.5.2. Moving object segmentation

Segmentation of video objects consists of separating the foreground objects from the background in a video [START_REF] Lee | Key-segments for video object segmentation[END_REF], [START_REF] Ochs | Higher order motion models and spectral clustering[END_REF]), (T. [START_REF] Wang | Probabilistic Motion Diffusion of Labeling Priors for Coherent Video Segmentation[END_REF]. It is important for a wide range of advanced video applications, including video content encoding [START_REF] Sikora | The MPEG-4 video standard verification model[END_REF], [START_REF] Zhang | Segmentation of moving objects in image sequence: A review[END_REF], video summary, annotation and video search, providing a spatial support for learning models of object class [START_REF] Prest | Learning object class detectors from weakly annotated video[END_REF], video surveillance, estimation of object movements, tracking, description of multimedia content [START_REF] Tekalp | Digital Video Processing[END_REF], [START_REF] Ebrahimi | MPEG-4 video verification model: A video encoding/decoding algorithm based on content representation[END_REF] intelligent signal processing, recognition of objects and activities, recognition of action [START_REF] Gorelick | Actions as Space-Time Shapes[END_REF]).

An object segmentation algorithm classifies the pixels of a video image into a number of classes that are homogeneous with respect to a few features.

The segmentation of an object is, therefore, an active research domain that has produced a wide variety of segmentation methods. Some methods focus on, depth information (3D movements) like [START_REF] Klappstein | Moving Object Segmentation Using Optical Flow and Depth Information[END_REF], [START_REF] Wedel | Detection and Segmentation of Independently Moving Objects from Dense Scene Flow[END_REF] and [START_REF] Herbst | RGB-D flow: Dense 3-D motion estimation using color and depth[END_REF], others group the points tracked during image pairs [START_REF] Brox | Object Segmentation by Long Term Analysis of Point Trajectories[END_REF], [START_REF] Ochs | Object segmentation in video: A hierarchical variational approach for turning point trajectories into dense regions[END_REF] or triplets [START_REF] Ochs | Higher order motion models and spectral clustering[END_REF], long-term trajectories [START_REF] Ochs | Segmentation of Moving Objects by Long Term Video Analysis[END_REF]. Some methods seek the infrastructure of different segments and shapes of the object in order to separate the segments of that object from the bottom [START_REF] Papazoglou | Fast Object Segmentation in Unconstrained Video[END_REF], [START_REF] Zhang | Video Object Segmentation through Spatially Accurate and Temporally Dense Extraction of Primary Object Regions[END_REF], [START_REF] Lee | Key-segments for video object segmentation[END_REF], and background subtraction [START_REF] Piccardi | Background subtraction techniques: a review[END_REF].

Recent approaches have used many methods of deep learning for object segmentation in videos: Some were based on a recurrent neural network (RNN)(Y.-T. [START_REF] Hu | MaskRNN: Instance Level Video Object Segmentation[END_REF], on CNNs [START_REF] Maninis | Convolutional Oriented Boundaries[END_REF], on optical flow [START_REF] Khoreva | Lucid Data Dreaming for Object Tracking[END_REF], on LSTM (N. [START_REF] Xu | YouTube-VOS: Sequence-to-Sequence Video Object Segmentation[END_REF], Fully Convolutional Networks (FCNs) (OSVOSS) [START_REF] Maninis | Video Object Segmentation Without Temporal Information[END_REF], on R-CNN [START_REF] Girshick | Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[END_REF], Mask R-CNN [START_REF] He | Mask R-CNN[END_REF], on modified Faster-RCNN [START_REF] Ren | Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[END_REF], and on Encoder-Decoder Seg-Net [START_REF] Badrinarayanan | SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation[END_REF].

VIII.5.3. Object classification

Object classification is the process of identifying what type of object is present in the environment among different available ones; for instance, to tell whether the moving objects are humans, vehicles, animals, inert objects or others. Object classification could distinguish remarkable motion from those caused by specular reflections, moving clouds, or other dynamic occurrences. Some problems appear when the background may contain element features similar to the foreground objects, e.g., when a many persons are moving Table Inter-object features

S_bboxes_objs_diff

The bbox surface differences between the two objects, normalized by the frame surface ((S_bbox(obj1)-S_bbox(obj2)) / S_bbox(obj1) )*100/ frame_width * frame_height).

S1_objs_diff

The surface differences between the two objects, normalized by the frame surface (S1_obj(obj1)-S1_obj(obj2) / S_bbox(obj1))*100/ frame_width * frame_height)..

S2_objs _diff

The surface differences between the two objects, normalized by the objects surface mean ((S1_ob(obj1)-S1_obj (obj2))/ S1_obj(obj1) *100/).

P_objs_diff

The perimeter differences between the two objects, normalized by the object surface (P_obj(obj1)-P_obj(obj2))*100/ (S1_ob(obj1)+S1_obj (obj2)/2).

Dist_objs

The relative distance between the two object centroids normalized by the frame surface (d(C(obj1), C(obj2))*100/ frame_width*frame_height).

Angle1_objs

The angle formed by the two objects vectors (

C(obj1) f-1 C(obj1) f → and C(obj2) f-1 C(obj2) f → ).

Angle2_objs

The angle formed by the two objects vectors (

Angle3_objs

The angle formed by the two objects vectors (

C(obj1) f-1 C(obj2) f-1 → and C(obj1) f C(obj2) f → ).

Angle4_objs

The angle formed by the two objects vectors (

Hu_objs_diff

The Hu invariant moments vector difference (7 moments) of the two object (sum(abs(Hu(obj1)-Hu(obj2))).

Huα_ objs_diff

The Hu invariant moments α difference of the two objects (Huα(obj1)-Huα(obj2)), where α є [1,…,7].

Form_objs_diff1

The difference between the two objects forms normalized by the objects surface mean, in a way similar to Form_change1(obj).

Form_objs_diff2

The upper part only of the difference between the two objects forms normalized by the objects surface mean, in a way similar to Form_change2(obj).

Objs_width_diff

The normalized difference between objects widths ((Obj_width(obj1) -Obj_width(obj2))/Obj_width(obj1)) *100/frame_width.

Objs_height_diff

The normalized difference between objects heights ((Obj_height(obj1) -Obj_height(obj2))/ Obj_height(obj1)) *100/frame_height.

Int_mean_objs_diff

The normalized difference of the objects intensity means ((Int_mean(obj1)-Int_mean(obj2))*100/ Int_mean(obj1)).

Int_std_objs_diff

The normalized difference of the objects intensity standard deviations ((Int_std(obj1)-Int_std(obj2))*100/ Int_std(obj1)). RGB_mean_objs_diff The normalized difference of the objects RGB means ((RGB_mean(obj1)-RGB_mean(obj2))*100/ RGB_mean(obj1)).

RGB_std_objs_diff

The normalized difference of the objects RGB standard deviations ((RGB_std(obj1)-RGB_std(obj2))*100/ RGB_std(obj1)).

D. Inter-Frames Features:

We take a window of M frames. In this window, and for each frame, we extract:

-The spatial features of object one and object two -The temporal features of object one and object two -The inter-objects features for object one and object two

For almost each of the features, we extract the derivative and second derivative, if it can be applied. Then for each of the features (f), its derivative (df) and its second derivative (ddf), we find seven global inter-frames features as follow in Table VIII-16.

Table VIII-16: Inter-frames features

Min/Max

The minimum value of the f/df/ddf between the window frames, normalized by the maximum value between the window frames.

First/Max

The value of the f/df/ddf in the first frame of the window frames, normalized by the maximum value between the window frames.

Last/Max

The value of the f/df/ddf in the last frame of the window frames, normalized by the maximum value between the window frames.

Middle/Max

The value of the f/df/ddf in the middle frame of the window frames, normalized by the maximum value between the window frames.

Average/Max

The average value of the f/df/ddf vector of all the window frames, normalized by the maximum value between the window frames.

Median/Max

The median value of the f/df/ddf vector of all the window frames, normalized by the maximum value between the window frames.

STD/Max

The standard variation value of the f/df/ddf vector of all the window frames, normalized by the maximum value between the window frames.

E. Trajectory Features:

The last set of features is related to trajectory. For that, we compute the trajectory of object one and object two centroids through window frames, and the trajectory of the middle points between object one and object two centroids through window frames. For each of those three trajectories, we apply three smoothing filters: -Average filter: This filter generates new trajectory points, where each point is the average between the precedent and the subsequent one. The goal here is to smooth rough trajectories obtain at the previous step.

VIII.8. Appendix 8: Sample of a CCTV report

In the Figure VIII-17, a sample of police reports issued from Beirut CCTV control room is shown, describing an incident of attack from a person on another. Where, in this incident report, tells that a time X date Y in Beirut location Z GPS (X,Y), on the right middle of the camera field of view, a person 1 enters the scene, having middle size (and clothes descriptions), having slow speed, heading south near the intersection XX. Also, similar description for another person 2 entering the camera field of view, where at a specific time, he slow down its speed, heading immediately north toward the person 1. At specific time he approaches to person 1, performing aggressive moves with his hands. Finally, at specific time, merge with person 1, and hit him on his head with black small object (physical interaction), person 1 fall down, person 2, heading south, on the down left of the camera field of view, increasing its speed, and exits the scene from street YY.

In this report, the four W's and the How are all answered. We highlighted in Figure VIII-17 some key words, as follow: Time and date (1), location and GPS (2), position in the frame, position to an area of interest and direction (3), physical description (size, closes colours,… ) of person 1 & 2 (4), speed ( 5), distance, and position and direction compared to the other person [START_REF] Klappstein | Moving Object Segmentation Using Optical Flow and Depth Information[END_REF], interaction [START_REF] Xu | Video structured description technology based intelligence analysis of surveillance videos for public security applications[END_REF], and object 1 and object 2 (8).

It is clearly noticed the similarity of these key words with our description.

  Abstract ...................................................................................................................................................

:

  The full description of the scene "LeftBox", results of mapping the M sac into the proposed templates. ......................................................................................................................... Table IV-5: The short description of the scene "LeftBox", describe at each moment only the corresponding irregularity. ............................................................................................... Table IV-6: Tables of datasets used ....................................................................................................

  Figure I-1: An example of video captioning architecture using sequence learning approach, taken from (Z. Wu, Yao, Fu, & Jiang, 2017) .................................................................................

Figure I- 2 :

 2 Figure I-2: An example of dense video captioning taken from (Zhou, Zhou, Corso, Socher, & Xiong, 2018). The colour bars represent different events. Coloured texts highlight relevant content to the event. ........................................................................................................

Figure II- 1 :

 1 Figure II-1: VSSD ontology's six main classes: Object, Video, Context, Activity, Scene and Descriptor. ...........................................................................................................................................

Figure II- 2 :

 2 Figure II-2: The Context class. ...............................................................................................................

Figure II- 3 :

 3 Figure II-3: The Object class. .................................................................................................................

Figure

  Figure II-4: The Video, object, video_object, and context classes. .......................................................

Figure II- 5 :

 5 Figure II-5: The Sub_object, Video_Object_state, and Sub_object_state classes. ...............................

Figure II- 6 :

 6 Figure II-6: The Activity, Action-Interaction, and Operation classes ....................................................

Figure II- 7 :

 7 Figure II-7: The Scene, and Descriptor classes ......................................................................................

Figure II- 8 :

 8 Figure II-8: Abstract description, having in the location and direction: U (Up), M (Middle), D (Down), R (Right), L (Left), I (inside), and O (outside). ....................................................................

Figure II- 9 :

 9 Figure II-9: The proposed "Video-Surveillance-Description Ontology". ...............................................

Figure III- 1 :

 1 Figure III-1: Symmetric Transfer Error ..................................................................................................

Figure III- 2 :

 2 Figure III-2: Symmetric Epipolar Distance .............................................................................................

Figure III- 3 :

 3 Figure III-3: Flow chart for the proposed method ................................................................................

Figure III- 4 :

 4 Figure III-4: Scene with a person (Frame 25): Marzat's algorithm applied for estimating motion. .....

Figure III- 5 :

 5 Figure III-5: Walking Scene: Frames references (Figure III-5-a and Figure III-5-b), and result of Marzat's algorithm applied give Figure III-5-c (without filtering). ....................................

Figure III- 6 :

 6 Figure III-6: Walking Scene: Uniformity Filtering result (regular size) of Figure III-5-c; we can notice that the groups of false vectors on the left and near the boy are deleted. ......................

Figure III- 7 :

 7 Figure III-7: Walking Scene: Texture Filtering (zoomed size) of Figure III-6; we can notice that the groups of false vectors near the left foot of the boy are deleted. ....................................

Figure III- 8 :

 8 Figure III-8: Highway 4 scene: Frames references (Figure III-8-a and Figure III-8-b), and result of Marzat's algorithm applied give Figure III-8-c (without filtering). .................................... Figure III-9: Highway 4 scene: Uniformity Filtering result (zoomed size) of Figure III-8-c; we can notice that the false vectors around the car are deleted. ...........................................................

Figure

  Figure III-10: Highway 4 scene: Texture Filtering result (zoomed size) of Figure III-9; we can notice that the false vectors near right doors of the car are deleted ..........................................

Figure III- 11 :

 11 Figure III-11:Scene with a person (Frame 25): Marzat's algorithm after filtering. ...............................

Figure III- 12 :

 12 Figure III-12: The case N 2 <=N/2. ...........................................................................................................

Figure III- 13 :

 13 Figure III-13: The case N 2 >N/2 ..............................................................................................................

Figure III- 14 :

 14 Figure III-14: Scene from "Highway 2": (a) Frame 97, (b) Frame 96, (c) Motion vectors, (d) Zoomed motion vectors (755×2 corresponding points). .................................................................

Figure III- 15 :

 15 Figure III-15: Scene from "Walking": (a) Frame 83, (b) Frame 80, (c) Motion vectors, (d) Zoomed motion vectors (365×2 corresponding points). .................................................................

Figure III- 16 :

 16 Figure III-16: Scene from "Bomb2": (a) Frame 117, (b) Frame 114, (c) Motion vectors, (d) Zoomed motion vectors (365×2 corresponding points). .................................................................

Figure III- 17 :

 17 Figure III-17: Ideal threshold (the yellow vertical line) and the discovered one (the violet oblique line). ...................................................................................................................................

Figure

  Figure III-18: Graph for F 2 with a mapping error threshold 𝛾′𝐹2 = 1, y=70, intersecting with Series 0 at 62.6, and Series 1 at 97. ................................................................................................

Figure

  Figure III-19: Graph of the H 1 with 𝜸′𝑯𝟏 = 𝟏 ......................................................................................

Figure III- 20 :

 20 Figure III-20: Graph of the F 1 with 𝜸′𝑭𝟏 = 𝟏 ........................................................................................

Figure III- 21 :

 21 Figure III-21: Graph of the H 2 with 𝜸′𝑯𝟐 = 𝟏 ......................................................................................

Figure III- 22 :

 22 Figure III-22: Graph of the F 2 with 𝜸′𝑭𝟐 = 𝟏 ........................................................................................

Figure III- 23 :

 23 Figure III-23: Graph of the H 3 with 𝜸′𝑯3 = 𝟏 .....................................................................................

Figure III- 24 :

 24 Figure III-24: Graph of the F 3 with 𝜸′𝑭𝟑 = 𝟏 ......................................................................................

Figure III- 25 :

 25 Figure III-25: Graph of the variation of curves with intersection points according to variable mapping error thresholds, for F, normalization 3, mean distance, and a mapping error threshold of 𝛾′𝐹3 = 0.2: 0.2 ∶ 10. ....................................................................................................

Figure IV- 1 :

 1 Figure IV-1: Proposed approach diagram, where each phase connected to an arrow starting point feeds the phase connected to the terminal point of the corresponding arrow. ..............

Figure IV- 2 :

 2 Figure IV-2: Objects movements and trajectories, Fight_RunAway2 scene from CAVIAR Video Sequence ("CAVIAR," 2004). .............................................................................................

Figure IV- 3 :

 3 Figure IV-3: Example: background Model (left image), activity map (right image), part of video Fight_OneManDown ("CAVIAR," 2004). ...........................................................................

Figure IV- 4 :

 4 Figure IV-4: Pixels quantification of activity map (left image), and morphological filters result (right image), part of video Fight_OneManDown ("CAVIAR," 2004). ......................................... Figure IV-5: Coloured morphological filters result (left image), and projected to background model (right image), part of video Fight_OneManDown. ............................................................

Figure IV- 6 :

 6 Figure IV-6: Coloured morphological filters result (left image), and projected to background model (right image) of scene "Fight_RunAway2". ....................................................................... Figure IV-7: Simple video segmentation according to the number of objects in the scene .................

Figure IV- 8 :

 8 Figure IV-8: Features extracted from a window of N frames. Each set of features at an arrow starting point feeds the set of features at its terminal point. ........................................................

Figure IV- 9 :

 9 Figure IV-9: Example scene left bag (left figure a), on the right side: X axis is the frame's number, Y axis is the percentage of changed pixels between 𝐼𝑥, 𝑦, 𝑡. 𝑀𝑡𝑎𝑥, 𝑦, 𝑡 𝑎𝑛𝑑 𝑀𝑡𝑏𝑥, 𝑦, 𝑡. 𝑀𝑡𝑎𝑥, 𝑦, 𝑡....................................................................

Figure IV- 10 :

 10 Figure IV-10: Extracting the scene classification vector indicating the existing or not of interaction. .........................................................................................................................................

Figure

  Figure IV-11: Objects Hu moment variations in the scene "LeftBox" from the database ("CAVIAR," 2004), it shows a brutal variations between frames 300 and 350, this is due to false detection. ........................................................................................................................

Figure IV- 12 :

 12 Figure IV-12: Objects surfaces variations in the scene "LeftBox", same false detection of the object 1 influence its surface. .......................................................................................................

Figure

  Figure IV-13: Left image shows the eight directions (in red and green) and the sixteen areas (dashed lines are the sectors borders). Right image shows trajectories (object1 trajectory in red, and object2 trajectory in green. ......................................................................................

Figure

  Figure IV-14: Figure showing the routes in green and the 2 levels of activity hot spots in blue and red, of scene "LeftBox. ............................................................................................................

Figure IV- 15 :

 15 Figure IV-15: Figure showing labelled route as A (green area) and activity hot spots as B (bleu area) and C (red area) in the scene "LeftBox". .........................................................................

Figure IV- 16 :

 16 Figure IV-16: Objects speed variations (skipping 10 frames between 2 positions used for computations) in the scene "LeftBox". ...........................................................................

Figure IV- 17 :

 17 Figure IV-17: Objects angles variations (skipping 25 frames between 2 positions used for computations) in the scene "LeftBox ". ..........................................................................

Figure

  Figure IV-18: Two Objects distances (skipping 5 frames between 2 positions used for computations) in the scene "LeftBox". ....................................................................................................

Figure IV- 19 :

 19 Figure IV-19: Interaction existence classification results for the scene "LeftBox", (0 no-interaction, 1 interaction): a-classification direct results, b-results after quantification (>=0.5) without filtering. ...........................................................................................................................

Figure IV- 21 :

 21 Figure IV-21: Interaction existence classification variance for the scene "LeftBox" after filtering ....

Figure IV- 22 :

 22 Figure IV-22: Classification direct results: blue line (physical (1) or distant (0)), orange line (peaceful (0) or aggressive (1)). Obviously all values are very close to 0 (distant and peaceful interaction). ..................................................................................................................... Figure IV-23: Objects accelerations (skipping 10 frames between 2 positions used for computations) in the scene "LeftBox": different threshold should be chosen for each of the objects. .

Figure

  Figure IV-24: Objects in the scene "LeftBox" at the frame 101: showing the trajectory of object 1, when the object 2 first enters the scene. .......................................................................

Figure IV- 25 :

 25 Figure IV-25: Objects distant interaction in the scene "LeftBox" at the frame 145: showing the first distant peaceful interaction between objects 1 and 2....................................................

Figure IV- 26 :

 26 Figure IV-26: Objects after interaction in the scene "LeftBox", at the frame 190. ............................

Figure IV- 27 :

 27 Figure IV-27: Object 2 exiting the scene "LeftBox", at the frame 238. ...............................................

Figure IV- 28 :

 28 Figure IV-28: Object 1 in the scene "LeftBox" is miss detected. .........................................................

Figure IV- 29 :

 29 Figure IV-29: Object 1 exiting the scene "LeftBox". ............................................................................

Figure IV- 30 :

 30 Figure IV-30: Confusion matrix for the chosen AdaBoost algorithm ..................................................

Figure IV- 31 :

 31 Figure IV-31: Confusion matrix for the chosen DNN algorithm ..........................................................

Figure IV- 32 :

 32 Figure IV-32: Objects distant interaction in the scene "Fight_RunAway2": frame 279 showing the trajectories of each object, and the distant aggressive interaction between objects 1 (showing as 5) and 2 (showing as 6). ...............................................................................

Figure

  Figure IV-33: Objects physical interaction in the scene "Fight_RunAway2": frame 321 showing the physical aggressive interaction between objects 1 and 2. The two object are here detected as being one (object number 2 (showing as 6).). .............................................

Figure IV- 34 :

 34 Figure IV-34: Objects after interaction in the scene "Fight_RunAway2: frame 468 showing no more interaction between the object 1 (as 6) and object 2 (as 8). ..........................................

Figure IV- 35 :

 35 Figure IV-35: Objects exiting in the scene "Fight_RunAway2": frame 488 showing the object 1 (as 6) exiting the scene. ............................................................................................................

Figure IV- 36 :

 36 Figure IV-36: Variations of Hu moments in scene "Fight_RunAway2". ..............................................

Figure IV- 37 :

 37 Figure IV-37: Variations of Objects surfaces in the scene "Fight_RunAway2". ..................................

Figure IV- 38 :

 38 Figure IV-38: Objects trajectories in the scene "Fight_RunAway2": the red trajectory is the object1 centroid displacement and the green trajectory is for the object2. ...............................

Figure IV- 39 :

 39 Figure IV-39: Figure showing, in the scene "Fight_RunAway2", the areas of interests (routes in green and the activity hot spots in blue and red). ....................................................................

Figure IV- 40 :

 40 Figure IV-40: Objects accelerations (skipping 5 frames between 2 positions used for computations) in the scene "Fight_RunAway2" for each of the two objects. ............................................

Figure IV- 41 :

 41 Figure IV-41: Objects angles variations (skipping 10 frames between 2 positions used for computations) in the scene "Fight_RunAway2". ............................................................

Figure IV- 42 :

 42 Figure IV-42: Two Objects distances in the scene "Fight_RunAway2": the local minimum occurs when the two objects approaches physically. The algorithm detects them as being one single object. The distance is then 0. .........................................................................................

Figure VIII- 1 :

 1 Figure VIII-1: Graphical representation of probability of 𝒂𝒓𝒈 𝒎𝒂𝒙𝑵, 𝑵𝟐𝑷[𝑿 ≥ 𝑵𝟐 for 1<= N <=40 and 1<= N 2 <=N, having p = 82.58, q= 17.42, f, Symmetric epipolar distance, normalization level 2, 𝜸𝑭𝟐 =2.2, and 𝜹𝑭𝟐 = 80.16. ........................................................

Figure VIII- 2 :

 2 Figure VIII-2: Scene of "Stairwell": a-frame 31, b-frame 28, c-motion's vectors, and d-motions' vectors zoomed (128*2 corresponding points) ..............................................................

Figure VIII- 3 :

 3 Figure VIII-3: Scene of "Stairwell": a-frame 44, b-frame 40, c-motion's vectors, and d-motions' vectors zoomed (293*2 corresponding points) ..............................................................

Figure VIII- 4 :

 4 Figure VIII-4: Scene "Stairwell": Percentage of correctly mapped points for H .................................

Figure VIII- 5 :

 5 Figure VIII-5: Scene of " Running ": a-frame 28, b-frame 27, c-motion's vectors, and d-motions' vectors zoomed (351*2 corresponding points) ..............................................................

Figure VIII- 6 :

 6 Figure VIII-6: Scene of " Highway 2 ": a-frame 97, b-frame 96, c-motion's vectors, and d-motions' vectors zoomed (690*2 corresponding points) ..............................................................

Figure VIII- 7 :

 7 Figure VIII-7: Scene of " Bomb 2 ": a-frame 117, b-frame 114, c-motion's vectors, and d-motions' vectors zoomed (317*2 corresponding points) ..............................................................

Figure VIII- 8 :

 8 Figure VIII-8: Two objects states in a frame. .......................................................................................

Figure VIII- 9 :

 9 Figure VIII-9: Figure showing bbox, mask and pixels features extracted from an object ...................

Figure

  Figure VIII-10: Temporal features of objects ......................................................................................

Figure

  Figure VIII-11: Figure showing the angle feature for object movement ............................................

Figure VIII- 12 :

 12 Figure VIII-12: Figure showing bbox, mask and pixels features extracted from an object .................

Figure

  Figure VIII-13: Inter-Objects features. ................................................................................................

Figure VIII- 14 :

 14 Figure VIII-14: Objects Trajectories .....................................................................................................

Figure VIII- 15 :

 15 Figure VIII-15: Original trajectories are in blue. The new trajectories after filtering are in red. a-Average filter, b-First order prediction filter, and c-Second order prediction filter .....

Figure VIII- 16 :

 16 Figure VIII-16: Graph representation of "Object characteristics template". Clauses in red indicate that the clause may not be present in the output sentence (to be decided according to rules). ...............................................................................................................................

Figure

  Figure VIII-17: A sample of a police CCTV report, with highlights on key words. ...............................

  Figure I-1, where 2D or 3D CNNs are exploited, on a video sequence, to extract features on optical flow images, video frames, or others… The video-level representations are then produced by mean pooling or soft attention over these visual features. Then, on the level of representations, an LSTM is trained for generating a sentence.

Figure

  Figure I-1: An example of video captioning architecture using sequence learning approach, taken from (Z.[START_REF] Wu | Deep Learning for Video Classification and Captioning[END_REF]

Figure I- 2 :

 2 Figure I-2: An example of dense video captioning taken from[START_REF] Zhou | End-to-End Dense Video Captioning with Masked Transformer[END_REF].The colour bars represent different events. Coloured texts highlight relevant content to the event.

Figure

  Figure II-1: VSSD ontology's six main classes: Object, Video, Context, Activity, Scene and Descriptor.

Figure

  Figure II-2: The Context class.

Figure

  Figure II-3: The Object class.

Figure

  Figure object, video_object, and context classes.

Figure II- 5 :

 5 Figure The Sub_object, Video_Object_state, and Sub_object_state classes.

Figure II- 6 :

 6 Figure II-6: The Activity, Action-Interaction, and Operation classes

Figure II- 7 :

 7 Figure II-7: The Scene, and Descriptor classes

Figure II- 8 :

 8 Figure II-8: Abstract description, having in the location and direction: U (Up), M (Middle), D (Down), R (Right), L (Left), I (inside), and O (outside).

Finally

  , in Figure II-9 we present all mentioned components of the "Video-Surveillance-Description Ontology".

Figure

  Figure II-9: The proposed "Video-Surveillance-Description Ontology".

Figure

  Figure III-1: Symmetric Transfer Error If we suppose that 𝑑(x ′ i , 𝐻. x i ) is the Euclidean image distance in the second image between the measured point x′ i and the point 𝐻. x i , and 𝑑(x i , 𝐻 -1 . x′ i ) is the distance in the first image. Then, the error for a couple of corresponding points x i ↔ x′ i can be measured in both images by a simple method called Symmetric Transfer Error (Figure III-1): 𝐸𝑟 𝑖 𝐻 = 𝑑(𝑥′ 𝑖, 𝐻. 𝑥 𝑖 ) 2 + 𝑑(𝑥 𝑖, 𝐻 -1 . 𝑥′ 𝑖 ) 2

  𝑖 𝑇 . 𝐹. 𝑥 𝑖 = 𝜀 ≠ 0 / 𝑖 = 1, … , 𝑁 / 𝜀 = algebraic error.(eq. III-5)Then the points x i and x′ i do not necessarily lies on the epipolar lines l' and l (see the FigureIII-2).

Figure

  Figure III-2: Symmetric Epipolar Distance

Figure

  Figure III-3: Flow chart for the proposed method

Figure

  Figure III-4: Scene with a person (Frame 25): Marzat's algorithm applied for estimating motion.

  Figure III-5: Walking Scene: Frames references (Figure III-5-a and Figure III-5-b), and result of Marzat's algorithm applied give Figure III-5-c (without filtering).

Figure III- 6 :

 6 Figure III-6: Walking Scene: Uniformity Filtering result (regular size) of Figure III-5-c; we can notice that the groups of false vectors on the left and near the boy are deleted.

Figure III- 7 :

 7 Figure III-7: Walking Scene: Texture Filtering (zoomed size) of Figure III-6; we can notice that the groups of false vectors near the left foot of the boy are deleted.

Figure III- 9 :

 9 Figure III-9: Highway 4 scene: Uniformity Filtering result (zoomed size) of Figure III-8-c; we can notice that the false vectors around the car are deleted.

Figure

  Figure III-10: Highway 4 scene: Texture Filtering result (zoomed size) of Figure III-9; we can notice that the false vectors near right doors of the car are deleted

  𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑜𝑏𝑗𝑒𝑐𝑡𝑠 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑜𝑏𝑗𝑒𝑐𝑡𝑠 𝑡𝑜 𝑓𝑖𝑛𝑑 (eq. III-12) 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑜𝑏𝑗𝑒𝑐𝑡𝑠 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑜𝑏𝑗𝑒𝑐𝑡𝑠(eq.

Figure

  Figure III-13: The case N 2 >N/2

  Figure III-14: Scene from "Highway 2": (a) Frame 97, (b) Frame 96, (c) Motion vectors, (d) Zoomed motion vectors (755×2 corresponding points).

  Figure III-16: Scene from "Bomb2": (a) Frame 117, (b) Frame 114, (c) Motion vectors, (d) Zoomed motion vectors (365×2 corresponding points).

Figure

  Figure III-17: Ideal threshold (the yellow vertical line) and the discovered one (the violet oblique line).

  (5 × 100) 𝑠 ⁄ , and the 5 th element in D r will have the percentage (5 × 100) 𝑟 ⁄ .  A graph is constructed such that: o The X axis represents the Placement Percentage. o The Y axis represents percentage of correctly mapped points. o Series 1 represents the reverse sorted ND s elements (i.e., frames) in blue. o Series 0 represents the sorted D r elements (i.e., frames) in red.

Figure

  Figure shows the graph for a normalization level of 2, with the Fundamental matrix F 2, using the mean distance, and with a mapping error threshold 𝛾′ 𝐹 2 = 1.

Figure

  Figure III-18: Graph for F 2 with a mapping error threshold 𝛾′ 𝐹 2 = 1, y=70, intersecting with Series 0 at 62.6, and Series 1 at 97.

Figure

  Figure III-19: Graph of the H 1 with 𝜸′ 𝑯 𝟏 = 𝟏 Figure III-20: Graph of the F 1 with 𝜸′ 𝑭 𝟏 = 𝟏

Figure

  Figure III-25: Graph of the variation of curves with intersection points according to variable mapping errorthresholds, for F, normalization 3, mean distance, and a mapping error threshold of 𝛾′ 𝐹 3 = 0.2: 0.2 ∶ 10.

  , which are: region-based tracking, contour-based tracking, feature-based tracking, model-based tracking, hybrid tracking, optical flow-based tracking, predictionbased techniques.

Figure

  Figure IV-1: Proposed approach diagram, where each phase connected to an arrow starting point feeds the phase connected to the terminal point of the corresponding arrow.

Figure

  Figure IV-2: Objects movements and trajectories, Fight_RunAway2 scene from CAVIAR Video Sequence ("CAVIAR," 2004).

Figure

  Figure IV-3: Example: background Model (left image), activity map (right image), part of video Fight_OneManDown ("CAVIAR," 2004).

  Figure IV-4 to determine location types, from high to low intensity: marginal areas, regular routes, and hot spots (highly frequented locations). . 2-Morphological filters: Opening and closing operators are a pp lied on the area generated by the previous quantification process to produce a smoother map, see Figure IV-4. 3-Applying on background model: finally, to visualize the routes and hot spots, we simply project the results on the background model. The marginal areas, the routes and the activity hot spots are shown in different colours in the following examples.

Figure IV- 4 :

 4 Figure IV-4: Pixels quantification of activity map (left image), and morphological filters result (right image), part of video Fight_OneManDown ("CAVIAR," 2004).

Figure IV- 5 :

 5 Figure IV-5: Coloured morphological filters result (left image), and projected to background model (right image), part of video Fight_OneManDown.

Figure IV- 6 :

 6 Figure IV-6: Coloured morphological filters result (left image), and projected to background model (right image) of scene "Fight_RunAway2".

Figure IV- 5 ,

 5 Figure IV-6 shows the routes in green and the activity hot spots in blue and red, of scene "Fight_RunAway2", taken from the database "Caviar" ("CAVIAR," 2004). We notice in Figure IV-5, Figure IV-6 and many other examples, not shown here, that the red areas are mainly the road intersections, road turns and objects meeting areas.

  Figure IV-9: Example scene left bag (left figure a), on the right side: X axis is the frame's number, Y axis is the percentage of changed pixels between 𝐼(𝑥, 𝑦, 𝑡). 𝑀 𝑡𝑎 (𝑥, 𝑦, 𝑡) 𝑎𝑛𝑑 𝑀 𝑡𝑏 (𝑥, 𝑦, 𝑡). 𝑀 𝑡𝑎 (𝑥, 𝑦, 𝑡).

Figure

  Figure IV-10: Extracting the scene classification vector indicating the existing or not of interaction.

  , between the Figure IV-24 and Figure IV-29. These characteristics are: 1-Object characteristics:

6

 6 Brutal changes of Hu moments indicate a big change or irregularity in the shape of the object. See Figure IV-11. ii. The object surface is associated with the temporal feature "S1_objs_diff" from Appendix VIII.6 Here again, brutal changes of this feature may indicate that something is occurring on the object surface. See Figure IV-12.

Figure

  Figure variations in the scene "LeftBox" from the database("CAVIAR," 2004), it shows a brutal variations between frames 300 and 350, this is due to false detection.

Figure

  Figure Objects surfaces variations in the scene "LeftBox", same false detection of the object 1 influence its surface.

Figure

  Figure IV-13: Left image shows the eight directions (in red and green) and the sixteen areas (dashed lines are the sectors borders). Right image shows trajectories (object1 trajectory in red, and object2 trajectory in green.

Figure

  Figure IV-14: Figure showing the routes in green and the 2 levels of activity hot spots in blue and red, of scene "LeftBox.

Figure

  Figure IV-15: Figure showing labelled route as A (green area) and activity hot spots as B (bleu area) and C (red area) in the scene "LeftBox".

Figure IV- 16 :

 16 Figure IV-16: Objects speed variations (skipping 10 frames between 2 positions used for computations) in the scene "LeftBox".

Figure

  Figure frames between 2 positions used for computations) in the scene "LeftBox ".

:

  for these features, we take the results of the states' values analysis in section IV.3.7 to indicate the variance in three: first, interaction existence classification variations are shown in Figure IV-19, Figure IV-20, and Figure IV-21; second, distant vs physical interaction classification variations (Figure IV-22); and third, aggressive vs peaceful interaction classification variations (Figure IV-22).

Figure

  Figure IV-19: Interaction existence classification results for the scene "LeftBox", (0 no-interaction, 1 interaction): a-classification direct results, b-results after quantification (>=0.5) without filtering.

Figure

  Figure IV-18: Two Objects distances (skipping 5 frames between 2 positions used for computations) in the scene "LeftBox".

Figure

  Figure IV-21: Interaction existence classification variance for the scene "LeftBox" after filtering

Figure

  Figure frames between 2 positions used for computations) in the scene "LeftBox": different threshold should be chosen for each of the objects.

  Figure IV-11, Figure IV-12, Figure IV-16, Figure IV-17, and Figure IV-18 we chose the thresholds in a way to not exceed more than 5 key moments for each characteristic.

Figure

  Figure Objects in the scene "LeftBox" at the frame 101: showing the trajectory of object 1, when the object 2 first enters the scene.

Figure

  Figure Objects distant interaction in the scene "LeftBox" at the frame 145: showing the first distant peaceful interaction between objects 1 and 2.

Figure

  Figure IV-26: Objects after interaction in the scene "LeftBox", at the frame 190. Figure IV-27: Object 2 exiting the scene "LeftBox", at the frame 238.

Figure IV- 28 :

 28 Figure IV-28: Object 1 in the scene "LeftBox" is miss detected. Figure IV-29: Object 1 exiting the scene "LeftBox".

Multi

  Finally, for each of the three classifications, one classification model, showing ultimate results, were selected. -For the Deep Learning Neural Networks experimentations: a multi-layered Deep Neural Network DNN was implemented for each classification. Simple Feedforward fully connected networks called Pattern recognition networks ("Pattern recognition network -MATLAB," n.d.) in MATLAB and Simulink environment were trained by back propagation of error. A standard network for pattern recognition is a two-layer feedforward network, with a sigmoid transfer function in the hidden layer, and a softmax transfer function in the output layer. In order to achieve the desired outputs, several tests were made after altering this model to handle deeper architecture (by adding more layers) and by determining the best architecture (number of layers, number of neurons) and the best parameters which maximise the results such as: 1) The activation function (Log-sigmoid 'logsig', positive linear function 'poslin' which is similar to 'ReLu', and hyperbolic tangent sigmoid function 'tansig') 2) The training method (Scaled conjugate gradient backpropagation 'trainscg', Levenberg-Marquardt backpropagation 'trainlm', Gradient descent with momentum and adaptive learning rate backpropagation 'traingdx', Gradient descent with momentum backpropagation 'traingdm', Gradient descent with adaptive learning rate backpropagation 'traingda', Conjugate gradient backpropagation with Polak-Ribiére updates 'traincgp', Conjugate gradient backpropagation with Fletcher-Reeves updates 'traincgf', Conjugate gradient backpropagation with Powell-Beale restarts 'traincgb', BFGS quasi-Newton backpropagation 'trainbfg', Resilient backpropagation 'trainrp', and One-step secant backpropagation 'trainoss'). 3) Lambda 4) Sigma.

  . As result of training this algorithm and testing it with 14902 records and 2305 features, having a training set of approximate 80% and a validation and test sets approximate 10 % each one, the accuracy of the test set achieved 87.5% after 325 epochs, see the confusion matrix at Figure IV-31.

Figure

  Figure IV-30: Confusion matrix for the chosen AdaBoost algorithm

Figure IV- 32 :

 32 Figure IV-32: Objects distant interaction in the scene "Fight_RunAway2": frame 279 showing the trajectories of each object, and the distant aggressive interaction between objects 1 (showing as 5) and 2 (showing as 6).

Figure

  Figure Objects physical interaction in the scene "Fight_RunAway2": frame 321 showing the physical aggressive interaction between objects 1 and 2. The two object are here detected as being one (object number 2 (showing as 6).).

Figure

  Figure Objects after interaction in the scene "Fight_RunAway2: frame 468 showing no more interaction between the object 1 (as 6) and object 2 (as 8).

Figure

  Figure IV-35: Objects exiting in the scene "Fight_RunAway2": frame 488 showing the object 1 (as 6) exiting the scene.

  moments or "Hu moments": In Figure IV-36, a simple threshold to 2 lead to localize seven peaks. Five of them are related with object 1 entering or leaving a sunny area, and where the object 1 is very far in the field of view. At that location, object 1 is miss-detected. The other two points indicate the two moments when the two objects are approaching physically (the algorithm detects them as one big object) and when they separate after the physical interaction.ii. The object surface: In Figure IV-37, putting threshold to 2 indicates three peaks.One of them corresponds to object 1 entering the sunny area. The other two peaks indicate the two moments when the two objects are approaching and separating after the physical interaction.

Figure

  Figure IV-36: Variations of Hu moments in scene "Fight_RunAway2".

Figure

  Figure IV-37: Variations of Objects surfaces in the scene "Fight_RunAway2".

Figure

  Figure Objects trajectories in the scene "Fight_RunAway2": the red trajectory is the object1 centroid displacement and the green trajectory is for the object2.

Figure

  Figure IV-39: Figure showing, in the scene "Fight_RunAway2", the areas of interests (routes in green and the activity hot spots in blue and red).

  c. Displacement related: i. The object speed variations: Figure IV-40 shows objects accelerations (skipping 5 frames between 2 positions used for computations) in the scene "Fight_RunAway2", where for each of the two objects the accelerations variations show two moments (when the two objects approaches and separates after the physical interaction and the algorithm detect them as one big object). Also variations for object 2 show big changes in the speed around the frame 240. ii. The object direction variations: Figure IV-41 shows objects angles variations (skipping 10 frames between 2 positions used for computations) in the scene "Fight_RunAway2", where, with a threshold fixed to 45 degrees this indicates six big changes. Two changes (1 and 3) indicate the two moments when the two objects approaches and separates after the physical interaction; Summits 2 indicate a big change in direction when objects are fighting. Two changes in blue (4 and 5) indicate the two moments (frames 243 and 462) when object 1 perform big change in its direction, and one change in orange (6) indicates when object 2 perform big change in its direction.

Figure

  Figure frames between 2 positions used for computations) in the scene "Fight_RunAway2" for each of the two objects.

Figure

  Figure IV-41: Objects angles variations (skipping 10 frames between 2 positions used for computations) in the scene "Fight_RunAway2".

Figure

  Figure Objects distances in the scene "Fight_RunAway2": the local minimum occurs when the two objects approaches physically. The algorithm detects them as being one single object. The distance is then 0.

  Figure VIII-2: Scene of "Stairwell": a-frame 31, b-frame 28, c-motion's vectors, and d-motions' vectors zoomed (128*2 corresponding points)

  Figure VIII-3: Scene of "Stairwell": a-frame 44, b-frame 40, c-motion's vectors, and d-motions' vectors zoomed (293*2 corresponding points)

Figure VIII- 4 :

 4 Figure VIII-4: Scene "Stairwell": Percentage of correctly mapped points for H

Figure VIII- 5 :

 5 Figure VIII-5: Scene of " Running ": a-frame 28, b-frame 27, c-motion's vectors, and d-motions' vectors zoomed (351*2 corresponding points)

Figure VIII- 6 :

 6 Figure VIII-6: Scene of " Highway 2 ": a-frame 97, b-frame 96, c-motion's vectors, and d-motions' vectors zoomed (690*2 corresponding points)

Figure VIII- 7 :

 7 Figure VIII-7: Scene of " Bomb 2 ": a-frame 117, b-frame 114, c-motion's vectors, and d-motions' vectors zoomed (317*2 corresponding points)

Figure

  Figure VIII-12: Figure showing bbox, mask and pixels features extracted from an object

Figure

  Figure VIII-13: Inter-Objects features.

Figure

  Figure

  Figure VIII-15-b. -Second order prediction filter: This filter generates new trajectory points p'', where each point is predicted from the position of the previous point on the trajectory, from the last speed vector, and is corrected with the last acceleration vector. See an example Figure VIII-15-c.

Table III -

 III 1: Table of ultimate thresholds: (a, b): a is the mapping error threshold, and b is the motion non-deformability threshold; below these thresholds is the corresponding percentage of success (%). ......................................................................................................................... Table III-2: Results from the temporal consistency-amelioration testing on 75 different videos (2141 frames), taking the Fundamental matrix, the Symmetric Epipolar distance, and the normalization level 2, where 𝛾𝐹2 =2.2 and 𝛿𝐹2 = 80.16. .................................................. Table III-3: Table of temporal motion non-deformability thresholds for each normalization when the mapping error threshold is fixed to 1 (𝛾′𝐹𝑛 = 1 and 𝛾′𝐻𝑛 = 1); (a, b): a is the percentage of success (%S), and b is the motion non-deformability threshold. ................................... Table IV-1: Templates used for video surveillance textual description ................................................ Table IV-2: Scene type classification into 15 classes according to number of objects before and after the action, background changes and object features of the moving objects.....................

Table IV-3: The scene activity characteristics matrix M sac showing objects and interaction states for the scene "LeftBox". ......................................................................................................... Table IV-4

Table IV -

 IV

7: Table listing 10 of the tested algorithms for objects segmentation and tracking ............ Table IV-8: Balanced dataset input characteristics. ............................................................................ Table IV-9: Used parameters for the three classification DNN algorithms ........................................ Table IV-10: The short description of the scene "Fight_RunAway2". ................................................

Table VIII -

 VIII 1: Comparison between the background subtraction, the temporal differencing, the optical flow approaches, block matching and feature correspondence. .....................................Table VIII-2: Numerical representation of probability of 𝑎𝑟𝑔 𝑚𝑎𝑥𝑁, 𝑁2𝑃[𝑋 ≥ 𝑁2 with p = 82.58 and q= 17.42, and N, N 2 = 1:15. ............................................................................................... Table VIII-3: Percentage of correctly mapped points (𝛾′𝐹𝑛 = 1 and 𝛾′𝐻𝑛 = 1) ............................... Table VIII-9: Percentage of correctly mapped points (𝛾′𝐹𝑛 = 1 and 𝛾′𝐻𝑛 = 1) ............................... Table VIII-10: Percentage of correctly mapped points (Normalization = 2, 𝛾𝐹2 =2.2, 𝛿𝐹2 = 80.16) . Table VIII-11: Percentage of correctly mapped points (𝛾′𝐹𝑛 = 1 and 𝛾′𝐻𝑛 = 1) ............................. Table VIII-12: Percentage of correctly mapped points (Normalization = 2, 𝛾𝐹2 =2.2, 𝛿𝐹2 = 80.16) . Table VIII-13: Spatial features of objects ............................................................................................ Table VIII-14: Temporal features of objects ....................................................................................... Table VIII-15: Inter-object features ..................................................................................................... Table VIII-16: Inter-frames features....................................................................................................

	Table VIII-4: Percentage of correctly mapped points (Normalization = 2, 𝛾𝐹2 =2.2, 𝛿𝐹2 = 80.16) ...

Table VIII-5: Percentage of correctly mapped points (𝛾′𝐹𝑛 = 1 and 𝛾′𝐻𝑛 = 1) ............................... Table VIII-6: Percentage of correctly mapped points (Normalization = 2, 𝛾𝐹2 =2.2, 𝛿𝐹2 = 80.16) ... Table VIII-7: Percentage of correctly mapped points (𝛾′𝐹𝑛 = 1 and 𝛾′𝐻𝑛 = 1) ............................... Table VIII-8: Percentage of correctly mapped points (Normalization = 2, 𝛾𝐹2 =2.2, 𝛿𝐹2 = 80.16) ...

  Video encoding stage: also known as Visual recognition where the visual features are directly extracted, more accurately, learnt, using different types of deep neural network algorithm, like Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) or Long Short-Term Memory (LSTM). The produced result composes a fixed or dynamic real-valued vector. b. Video decoding stage: also known as the sequence generation or text generation, where the vector result of the first stage is fed for text generation, as single or multiple sentences. For decoding, first RNN were used, RNN is a neural network adding extra feedback connections to feed-forward networks, enabling it to work with sequences of inputs. Then, the network is updated grounded on every input item and the preceding hidden state. They are networks with loops that allow persevering information. These networks, mainly, have been used in many fields such as speech recognition, language modelling, image captioning, translation and more. Different types of deep neural network are now in use, most commonly, deep RNN, Bi-directional RNN, Long Short-Term Memory LSTM, Gated Recurrent Units (GRU) or others.

  the Homography matrix H n ) to relate the normalized corresponding points (𝑥̅ 𝑖 In the section III.4, we describe several experiments and an intriguing method of searching for thresholds using a new type of graph-called the "Best Maximum -Acceptable Minimum Graph" (see Section 4). For each normalization (1 to 10), each transformation (Fundamental and Homography), and each type of error (mean distance, Symmetric Transfer Error, or the Symmetric Epipolar Distance), the ultimate couple mapping error threshold (γ F are found in a way that maximizes the success (the percentage of success) of the algorithm. The ultimate thresholds are shown the TableIII-1. Furthermore, we describe the processes in the algorithm (Subsection III.3.6), and some experiments and new methods of searching for ultimate thresholds in Section III.4.

			𝑛 ↔ 𝑥̅ ′ 𝑖 𝑛 /i= 1… N/
	n=1, 2, 3…). Then, for any normalization level, the percentage of correctly mapped points
	(p F known (γ F n for F, and p H n for F, and γ H n for H) is calculated using the mapping error threshold that is already n for H). Then, p F n ) is compared with the motion n (respectively, p H non-deformability threshold that is already known (δ F n for F and δ H n for H). If p F n >= δ F n , then
	the motion (i.e., temporal movement under testing) is non-deformable regarding F, and if
	p H respectively, when p H n >= δ H n , then the motion is non-deformable regarding H; otherwise, when p F n < δ H n ) the motion is deformable.	n < δ F	n (or
						n ,
	γ' F δ H n n for F and γ H , δ' H n for H)	n , γ' H	n for H) and the motion non-deformability threshold (δ F	n , δ' F	n for F and

Table III -

 III 1: Table of ultimate thresholds: (a, b): a is the mapping error threshold, and b is the motion nondeformability threshold; below these thresholds is the corresponding percentage of success (%).

	Normalization Transformation	1	2	3	4	5	6	7	8	9	10
	H	Mean distance	%𝑺: (𝜸′ 𝑯 𝒏 , 𝜹 ′ 𝑯 𝒏 ):	(1.4, 87.8)	(2.4, 84.07 )	(3.8, 85.66 )	(5, 85.28 )	(6.4, 85.84 )	(7.4, 84.65 )	(9, 85.29 )	(10.8, 86.06 )	(13, 86.69 )	(14, 85.11 )

73.42 73.57 73.85 74.32 74.29 74.23 75.03 72.87 71.53 73.9

	F	Mean distance	(𝛄′ 𝐅 𝐧 , 𝜹 ′ 𝑭 𝒏 ): %𝐒:	(0.6, 83.36 ) 81.56 82.16 (1, 79.13 )	(1.4, 76.92 ) 82	(1.8, 76.04 ) 82.05 82.04 82.32 82.93 82.68 80.98 80.51 (2.2, (2.8, (3.8, (4.8, (5, (6, 75.52 76.65 80.91 82.56 81.06 90.76 ) ) ) ) ) )
	H	Symmetri c Transfer	(𝛄 𝐇 𝐧 , 𝜹 𝑯 𝒏 ):	(3, 84.53 )	(12.6, 85.19 )	(28.4, 85.3)	(50, 85.28 )	(78, 85.16 )	(120, 85.43 )	(160, 85.14 )	(230, 85.8)	(345, 86.89 )	(350, 83.66 )
		Error	%𝑺:	73.86 74.48 74.62 74.32 74.51 74.23 75.33 72.75 71.63 74.15
	F	Symmetri c Epipolar	(𝛄 𝐅 𝐧 , 𝜹 𝑭 𝒏 ):	(0.6, 81.31 )	(2.2, 80.16 )	(3.8, 76.6)	(6.6, 76.25 )	(15, 81.31 )	(25, 83.08 )	(35, 83.02 )	(47, 82.61 )	(51, 81.26 )	(72, 81.18 )
		Distance	%𝑺:	81.8	82.58

82.09 82.41 82.79 82.92 83.12 82.64 81.45 82.44

  -2 above. When applied a second time, the percentage of success (percentage of true classification) increased to more than 91.8%. By

	𝐴𝑟𝑔 𝑚𝑎𝑥 𝑁,𝑁 2	𝑃[(𝑋 ≥ 𝑁 2 ) ∩ (𝑌 < 𝑁 2 )]	𝑎𝑟𝑔 𝑚𝑎𝑥 𝑁,𝑁 2 𝑎𝑟𝑔 𝑚𝑎𝑥 𝑁,𝑁 2	(∑ 𝑁 𝑖=𝑁 2 (∑ 𝑁 𝑖=𝑁 2	𝐶 𝑁 𝑖 𝑝 𝑖 𝑞 𝑁-𝑖 𝐶 𝑁 𝑖 𝑝 𝑖 𝑞 𝑁-𝑖	)(∑ 𝑁 2 -1 𝑖=0 )(∑ 𝑁-𝑁 2 𝑖=0	𝐶 𝑁 𝑖 𝑞 𝑖 𝑝 𝑁-𝑖 𝐶 𝑁 𝑖 𝑞 𝑖 𝑝 𝑁-𝑖 ) / If N2 ≤ N/2
		=

Table III -

 III 3: Table of temporal motion non-deformability thresholds for each normalization when the mapping error threshold is fixed to 1 (𝜸′ 𝑭 𝒏 = 𝟏 and 𝜸′ 𝑯

	Normalization Transformation	1	2	3	4	5	6	7	8	9	10
	H mean distance (%𝑺, 𝜹 ′ 𝑯 𝒏 ):	(73.28, 78.06)	(65.9, 43.61)	(64.49, 25)	(62.15, 15)	(61.46, 9.88)	(62.2, 6.74)	(61.36, 4.8)	(58.13, 3.4)	(58.63, 2.65)	(59.34, 1.94)
	F mean distance (%𝐒, 𝜹 ′ 𝑭 𝒏 ):	(79.95, 92.49)	(82.16, 79.13)	(80.45, 67.22)	(79.23, 57.87)	(77.57, 49.82)	(74.86, 42.16)	(72.95, 36.62)	(72.2, 31.47)	(71.08, 27.3)	(73.52, 23.21)

𝒏 = 𝟏); (a, b): a is the percentage of success (%S), and b is the motion non-deformability threshold.

Table IV -

 IV 1: Templates used for video surveillance textual description

	Reference	Template
	(Nishida & Takamatsu,
	1982), (Nishida, Takamatsu, Tani, & Doi,	A/the (agent) (verb) (object) (location)
	1988)
	(Ivanov et al., 1999)	(Agent) (verb) (frame range)
	(Gerber et al., 2002)	(Interval of validity)! (verb) phrase(subject, object)
	(Jiangung Lou et al.,
	2002)
	(Carles Fernández et al.,
	2008), (C. Fernández et	Verb (Agent, direction, location)
	al., 2011)
	(Fernández Tena, 2010)	(Frame number)! (Agent), (location in frame (x,y)), (direction), (velocity), (action)
	(Khan, Lei Zhang, &	(subject (S)) performs (action (A)) on (object (O))
	Gotoh, 2011)
		(subject)( verb) // (subject)( verb) (object) // (subject)( verb)
	(Barbu et al., 2012)	(Complement) //
		(subject) [adverb](verb)(object) [Person pose]
		A (Color) (heavy, medium, light) (Car, Bus, Cycle, Bike, vehicle)
	(Ahmed et al., 2019)	(moving, moving high-speed, moving low-speed) from (Region Name) toward (Region Name) // A Person (walking, running,
		loitering) from (Region Name) toward (Region Name)

(The Obj) (Action) in (The place name) [at (high/low/middle) speed]

Table IV -

 IV 2: Scene type classification into 15 classes according to number of objects before and after the action, background changes and object features of the moving objects

	Features to test	Number	Number	Background	Features changes of moving objects
			of objects	of objects	changes
			before	after	detection
	Type of scene	action	action
	1	0 object	NA	NA	NA	NA
	2	1 Object  0a	1	0	No
	3	1 Object  0b	1	0	Yes
	4	1 Object  1a	1	1	No
	5	1 Object  1b	1	1	Yes	No Hu moments (Hu_objs_diff) changes
						Hu moments (Hu_objs_diff) changes, and
						surface (S1_objs_diff) changes
	6	1 Object  1c	1	1	Yes	In plus, for Left object case: Hu moments or
						surface are smaller. For taken object case:
						Hu moments or surface are bigger
	7	1 Object  1c	1	1	Yes	All the moving object features indicate different object before and after
	8	1 Object  2a	1	2	Yes
	9	1 Object  2b	1	2	No
	10 2 Objects  1a	2	1	Yes
	11 2 Objects  1b	2	1	No
	12 2 Objects  0	2	0	Yes
	13 2 Objects  2a

Need more processing, see section IV.3.7 14 2 Objects  2b 15 Many Objects Many Many

Table IV -6: Tables of datasets used

 IV

									Number	Total	Total
									of scenes	duration	duration
	#	Name	Reference		Description	Main area of application	extracte d	of scenes extracted	of scenes taken
									(per sec)	(per sec)
		BEHAVE Test Case	Interactions	(Blunsden & Fisher, 2010)	This dataset comprises various scenarios of people interacting. It covers 10 types of interactions: in following, chase, fight, run together, and meet. group, approach, walk together, split, ignore,	Event detection, modelling recognition, visual tracking crowd, multi-objects activities	51	589	327
		CAVIAR: Context Aware Vision using Image-based Active Recognition	("CAVIAR," 2004)		This dataset contains many scenes type: Walking, fainting, Leaving bags behind, groups meeting, walking together and splitting up, two people fighting.	Interaction analysis, activity segmentation, tracking recognition, trajectory clustering of, motion	18	128	101
		"EPFL" data set: Multi-camera Pedestrian Videos	("EPFL," n.d.)		This dataset contains four sequences: 1-Laboratory Passageway sequence (underground train station). sequences (4 videos inside), Campus sequences (2 videos outside), Terrace sequences (outside),	people detection and tracking	29	248	125
						This dataset contains 20 video sequences (of 1 minute	Interaction analysis, Complex
		UT-Interaction dataset	(Ryoo et al., 2010)		each), showing 6 classes of human interactions:	activities recognition, Action	13	424	138
						shake-hands, point, hug, push, kick and punch.	recognition.
		Activity modelling and abnormality detection dataset	(Varadarajan Odobez, 2009)	&	This dataset consists of a 45 minutes long video of a junction controlled by traffic lights.	multi-object tracking	6	26	12
		Advanced Video and Signal based Surveillance	(i-LIDS, 2007)		This dataset consists of 3 sequences (abandoned baggage) and 4 sequences (parked vehicle) of approximately 20 minutes duration.	abandoned baggage detection, parked vehicle detection	11	125	24
		Audio-visual dataset	people	(Audiovisual people dataset, n.d.)	This dataset consists of three sequences with a video camera and two microphones.	people detection tracking	5	106	72
		VISOR Video surveillance online repository	("VISOR," 2017)		This dataset sequence presents 15 people walking on square, interacting and involving in different groups.	Repository, interaction analysis,	4	38	28
						This dataset consists of many real outdoor scenes	Activity	analysis,	tracking,
		VIRAT Video Dataset	(Oh et al., 2011)		including 23 event types distributed among numbers	human-vehicle	interaction	13	79	39
						of instances throughout 29 hours of video.	recognition, action recognition

Table IV -7: Table listing 10 of the tested algorithms for objects segmentation and tracking

 IV

	#	Algorithm	Reference	Tracking	Multi-object	Segmen tation	Object type	Free	Notes	Implement ation	Subjective quality
		Tracking Interacting Objects	(X. Wang, Türetken, Fleuret, & Fua,	Yes	yes	no	ANY	upon request ("Tracking Interacting Objects -CVLAB,"	Based on network-flow Mixed Integer Program	Moderate	++
			2016)					n.d.)
		Globally-Optimal Greedy Algorithms	(Pirsiavash, Ramanan, & Fowlkes, 2011)	yes	yes	no	Pedest rian	Yes, (Pirsiavash, n.d.)	Based on flow network	Very Hard	++
		Continuous Energy Minimization for Multi-Target Tracking	(Milan, Roth, & Schindler, 2014)	yes	yes	no	Pedest rian	Yes, (Milan, 2014)	Based on continuous energy minimization	Moderate	+
		Discrete-Continuous Energy Minimization for Multi-Target Tracking	(Andriyenko, Schindler, & Roth, 2012)	yes	yes	no	Pedest rian	Yes, (Milan, 2012)	Based on Discrete-Continuous energy minimization	Hard	+
		Two-Granularity Tracking	(Fragkiadaki, Zhang, Zhang, & Shi, 2012a)	yes	yes	yes	Pedest rian	Yes, (Fragkiadaki, Zhang, Zhang, & Shi, 2012b)	Based on Mediating Trajectory and Detection Graphs	Hard	+
		GMCP-Tracker	(Roshan Zamir, Dehghan, & Shah, 2012)	yes	yes	no	Pedest rian	Yes, (Roshan Zamir et al., 2012)	Global Multi-object Tracking Using the temporal domain to the object domain Generalized Minimum Clique Graphs Based, on shifting the approximation from	Moderate	+
		Urban Tracker	(Jean-Philippe Jodoin, Saunier, 2014) Bilodeau, &	yes	yes	no	ANY	Yes, (J.-P. Jodoin, 2013) Bilodeau, & Saunier,	Multiple Object Tracking in Urban Mixed detect moving objects Traffic, based on background subtraction to	Moderate	+
		Moving-Target-tracking-with-opencV		yes	yes	yes	Any	Yes, (Son, 2015/2019)	Based on Background subtraction and Kalman Filter	Moderate	+++
		Online Multi-Object Tracking by Decision Making	(Y. Xiang, Alahi, & Savarese, 2015a)	yes	yes	no	Any	Yes, (Y. Xiang, Alahi, & Savarese, 2015b)	Based on Markov decision processes	Moderate	+
		Motion-Based Multiple Object Tracking		yes	yes	yes	Any	Yes, ("Motion-Based Multiple Object Tracking -MATLAB & Simulink," n.d.)

  Table IV-8: Table IV-8: Balanced dataset input characteristics.

		Input records	Features per record	Input records classifications
	Interaction and non-interaction	14902	2305	Interaction	Non-interaction
	classification algorithm			7657	7245
	Distant and physical			Physical	Distant
	interaction classification algorithm	7079	2303	3640	3439
	Aggressive and peaceful			Aggressive	Peaceful
	interaction classification algorithm	6703	2303	3439	3264

Table IV -

 IV 9: Used parameters for the three classification DNN algorithms

		chosen	Number of	Number	Training	Sigma	Lambda	Regulari	Activation
		algorithm	layers	of	function	𝝈	λ	zation	function
				neuron
				s in
				each
				hidden
				layer
	Interaction or no interaction	Pattern recogn ition network:	7 (6 hidden, 1 output)	586	Trainscg	𝟓.𝟎 𝐞^(-𝟕)	𝟓.𝟎 𝐞^(-5)	0.5	Logsig
	Distant or physical interaction	feedforward network composed of fully	4 (3 hidden, 1 output)	426	Trainscg	𝟓.𝟎 𝐞^(-𝟕)	𝟓.𝟎 𝐞^(-7)	0.5	Logsig
	Aggressive or peaceful interaction	connected layers	4 (3 hidden, 1 output)	546	Trainscg	𝟓.𝟎 𝐞^(-𝟕)	𝟓.𝟎 𝐞^(-7)	0.5	Logsig

Table IV -

 IV 10: The short description of the scene "Fight_RunAway2".

	Info Frame # Desc target
		193	Object
		196	Object
			Object
			Object
			Object 1 & Object 2
			Object
			Object
			Object 1 & Object 2
			Object
			Object
			Object
			Object
			Object
			Object
			Object
			Object
			Object
			Object 1 & Object 2
			Object
			Object
			Object
			Object
			Object 1 & Object 2
			Object
			Object
			Object
			Object
			Object
			Object
			Object
			Object
			Object
	VIDEO ID	488 468	Object Object Object Object Object 1 & Object 2
		491	Object

  b) How to decide what visual information to extract from video? As we know what we want to use the system for, what to describe, and what we need to search for in the video archives, this can help to decide what visual features to extract.

  , Block matching techniques consist of three main components: block determination, search methods, and matching criteria. Block location, the size, and the scale can be determined by a simple or a hierarchical approach. There are several block-based motion estimation methods (search methods), to mention: Full Search Algorithm (FSA) or Exhaustive Search Algorithm (ESA), two dimensional Logarithmic Search (LOGS) , three-Step Search (3SS), four Step Search (4SS) , Adaptive Rood Pattern Search (ARPS), Diamond Search (DS), Modified Orthogonal Search Algorithm (MOSA), Cross Search Algorithm (CSA), Binary Search (BS), Hierarchical Search Algorithm (HSA), etc.

Table VIII -

 VIII 2: Numerical representation of probability of 𝒂𝒓𝒈 𝒎𝒂𝒙 𝑵,𝑵 𝟐 𝑷[(𝑿 ≥ 𝑵 𝟐 ) with p = 82.58 and q= 17.42, and N, N 2 = 1:15.

Table VIII -3: Percentage of correctly mapped points (𝜸′ 𝑭

 VIII

	Normalisation Transformation	1	2	3	4	5	6	7	8	9	10
	H mean dista nce F	n p H (𝜹 ′ 𝑯 𝒏 ): 78.06 43.61 85.15 75 n 99.21 96.87 p F (𝜹 ′ 𝑭 𝒏 ): 92.49 79.13 67.22 57.87 49.82 42.16 36.62 31.47 69 43.75 37.5 30.46 23.43 19.53 25 15 9.88 6.74 4.8 3.4 91.40 82.81 75 67.96 63.28 50	12.5 2.65 42.18 27.3	10.15 1.94 40.62 23.21
		Table VIII-4: Percentage of correctly mapped points (Normalization = 2, 𝜸 𝑭 𝟐 =2.2, 𝜹 𝑭 𝟐 = 80.16)

𝒏 = 𝟏 and 𝜸′ 𝑯 𝒏 = 𝟏)

Table VIII -5: Percentage of correctly mapped points (𝜸′ 𝑭

 VIII

	Normalisation Transformation	1	2	3	4	5	6	7	8	9
	H mean	p H n
	distanc
	e

𝒏 = 𝟏 and 𝜸′ 𝑯 𝒏 = 𝟏)

58.020 20.47 6.484 3.754 2.047 1.023 1.023 1.023 1.0239

		(𝜹 ′	𝑯 𝒏 ): 78.06	43.61	25	15	9.88	6.74	4.8	3.4	2.65
	F	n p F (𝜹 ′ 𝑭 𝒏 ): 92.49 80.54	51.53 79.13	35.83 67.22	27.64 57.87	23.20 49.82	19.11 42.16	16.72 36.62	15.01 31.47	11.945 27.3

Table VIII -6: Percentage of correctly mapped points (Normalization

 VIII

					= 2, 𝜸 𝑭 𝟐 =2.2, 𝜹 𝑭 𝟐
				= 80.16)
		Symmetric	𝒑 𝑭 𝒏	53.5836
	F	epipolar distance	(𝜹 ′	𝑭 𝟐 ):	80.16

Table VIII -7: Percentage of correctly mapped points (𝜸′ 𝑭

 VIII

	Normalisation Transformation	1	2	3	4	5	6	7	8
	H mean	n p H (𝜹 ′ 𝑯 𝒏 ): 78.06 56.849 32.191 43.61	21.461 25	11.415 15	7.9909 9.88	5.9361 6.74	4.3379 4.8	3.1963 3.4
	F	distance	p F n	80.

𝒏 = 𝟏 and 𝜸′ 𝑯 𝒏 = 𝟏)

593 55.9361 47.4886 37.8995 31.9635 27.1689 23.5160 20.7763

	(𝜹 ′	𝑭 𝒏 ):

92.49 79.13 67.22 57.87 49.82 42.16 36.62 31.47

	Table VIII-8: Percentage of correctly mapped points (Normalization =
	2, 𝜸 𝑭 𝟐 =2.2, 𝜹 𝑭 𝟐 = 80.16)
	F	Symmetric epipolar distance	𝒑 𝑭 𝒏 (𝜹 ′ 𝑭 𝟐 ):	57.5342 80.16

Table VIII -

 VIII

		Normalisation Transformation	1	2	3	4	5	6	7	8	9
	H	mean	p H n
		distance

9: Percentage of correctly mapped points (𝜸′ 𝑭 𝒏 = 𝟏 and 𝜸′ 𝑯 𝒏 = 𝟏)

91.390 88.344 85.827 58.807 32.053 23.708 19.337 16.158 12.715

	(𝜹 ′	𝑯 𝒏 ):

78.06 43.61

					25	15	9.88	6.74	4.8	3.4	2.65
	F	n p F (𝜹 ′ 𝑭 𝒏 ):	98.41	94.17	93.11	92.05	90.72	90.19	89.00	87.54	86.75

92.49 79.13 67.22 57.87 49.82 42.16 36.62 31.47 27.3 Table VIII-10: Percentage of correctly mapped points (Normalization

				= 2, 𝜸 𝑭 𝟐 =2.2, 𝜹 𝑭 𝟐 = 80.16)
	F	Symmetric epipolar distance	𝒑 𝑭 𝒏 (𝜹 ′ 𝑭 𝟐 ):	94.4371 80.16

 Scene of "Bomb 2" :

o Frame 117:

Table VIII -11: Percentage of correctly mapped points (𝜸′ 𝑭

 VIII

	Normalisation Transformation	1	2	3	4	5	6	7	8	9	10
	H	mean	p H n	92.
		distance

𝒏 = 𝟏 and 𝜸′ 𝑯 𝒏 = 𝟏)

42 71.29 48.58 33.12 23.97 18.92

								12.93	11.35 9.77	7.57
		(𝜹 ′	𝑯 𝒏 ): 78.06 43.61 25	15	9.88	6.74	4.8	3.4	2.65	1.94
	F	p F n	99.

053 94.6 84.22 79.4 75.39 71.92 65.9306 60.56 54.57 48.5

	(𝜹 ′	𝑭 𝒏 ):

92.49 79.13 67.22 57.87 49.82 42.16 36.62 31.47 27.3 23.21 Table VIII-12: Percentage of correctly mapped points (Normalization

				= 2, 𝜸 𝑭 𝟐 =2.2, 𝜹 𝑭 𝟐 = 80.16)
	F	Symmetric epipolar distance	𝒑 𝑭 𝒏 (𝜹 ′ 𝑭 𝟐 ):	95.2681 80.16

It is an estimation to linearize the least squares, because the calculations with least squares risk producing an absurd estimation. So the least squares: became:, with α adjustable, representing the regularity of the solution.

The case of a general projective camera as uncalibrated camera is the case of this study, seeking more generality.

The Direct Linear Transform (DLT) algorithm was introduced by Abdel-Aziz and Karara (Abdel-Aziz & Karara, 1971).

The normalization step was introduced by Hartley (R. I.[START_REF] Hartley | In defense of the eight-point algorithm[END_REF].

The motion (i.e., the temporary motion) is denoted according to the frame of its motion vectors and the destination frame. For example, the displacement of the object from frame Xk (the suitable corresponding frame of Xj for the study) to frame Xj (Xk  Xj) is called motion Xj.

Deformable/non-Deformable Motion Algorithm Deformable/non-Deformable Object Algorithm

Surface [having now {"Smaller" | "Bigger"} one], and having respectively [considerable | slight] {"Increasing" of its | "Decreasing" of its | "Stable"} speed}.

2-Inter-objects characteristics templates: two templates were introduced: a-When the second object enters the scene, at this key moment the description is activated and follows "Inter-Objects entrance template": {The two objects are respectively {"far" | "close"}, {no interaction | a {"distant" | "physical"} {"aggressive" | "peaceful"} interaction} occurs between them}.

b-At each of the key moments a proposed "Inter-Objects characteristics template" is structured as follow:

{The two objects are respectively {"approaching" | "receding" | "merged"}, {no interaction | a {"distant" | "physical"} {"aggressive" | "peaceful"} interaction} occurs between them}.

When an object first entrance occurs, we use indicative description (big surface, slow speed, regular shape, etc) of its state in the entrance template (see section II.3.6). Later, at a moment k m during the scene, we use comparative description (bigger surface, increasing the speed, etc) of the current moment k m state with the last state moment k m-1 .

To establish the mapping between the scene activity characteristics matrix M sac and the new proposed structured templates, simple logical threshold-based rules, founded on the ontology shown in chapter II, were used. Examples of these rules are shown as follow:

-For the position to an area of interest: we introduced three rules for the "Object characteristics template": a) For an object l at a key moment k m є { k 1 , k 2 , … k n }, in a given scene, if ∃ Å ∈ "area of interest" / 𝐶(𝑜𝑏𝑗 𝑙 (𝑘 𝑚 )) ∈ Å ⟹ 𝑎𝑑𝑑 𝑡𝑜 𝑑𝑒𝑠𝑐𝑟𝑖𝑝𝑡𝑖𝑜𝑛 {moves, in "Å" spot}. C(𝑜𝑏𝑗 𝑙 ) represents the centroid position of object l є { 1,2} . This rule indicates that if the object's centroid belongs to an area of interest, we map the area symbol into the template. b) For an object 𝑙 at a key moment 𝑘 𝑚 , in a given scene, if ∃ Å ∈ "area of interest" / 𝐶(𝑜𝑏𝑗 𝑙 (𝑘 𝑚 )) ∉ Å and 𝐶(𝑜𝑏𝑗 𝑙 (𝑘 𝑚-1 )) ∈ Å ⟹ 𝑎𝑑𝑑 𝑡𝑜 𝑑𝑒𝑠𝑐𝑟𝑖𝑝𝑡𝑖𝑜𝑛 {𝑙𝑒𝑎𝑣𝑒𝑠 "Å" 𝑠𝑝𝑜𝑡, 𝑎𝑛𝑑 𝑚𝑜𝑣𝑒𝑠 }. This rule indicates that if the current object's centroid does not belong to any of the areas of interest, but its position, at the past key moment, was belonging to any area, we map the symbol of the area into the template as, {leaves "area of interest" spot, and moves}. c) For an object 𝑙 at a key moment 𝑘 𝑚 , in a given scene, if Ɐ Å ∈ "area of interest" / 𝐶(𝑜𝑏𝑗 𝑙 (𝑘 𝑚 )) ∉ Å and 𝐶(𝑜𝑏𝑗 𝑙 (𝑘 𝑚-1 )) ∉ Å ⟹ 𝑎𝑑𝑑 𝑡𝑜 𝑑𝑒𝑠𝑐𝑟𝑖𝑝𝑡𝑖𝑜𝑛 {𝑚𝑜𝑣𝑒𝑠}.

VIII. Appendices

VIII.1. Appendix 1: related works in the domain of video surveillance ontologies

In addition to what was mentioned in the chapter II, other interesting ontologies in the video surveillance domain can be found in the literature, we mention:

-A work of detecting events and objects is presented by [START_REF] Kazi Tani | Events Detection Using a Video-Surveillance Ontology and a Rule-Based Approach[END_REF] which propose to use, together, an ontology with a rule detection system. The system uses probabilities to estimate certain events as well as its initial and final times. -Video event analysis ontology is presented in [START_REF] Sanmiguel | An Ontology for Event Detection and its Application in Surveillance Video[END_REF]. Their ontology is based on two levels of knowledge: the application domain (high level semantic concepts as objects, context, and events) and the analysis system (algorithms, reactions to events, etc.). The ontology and the case study are specialized for the Underground video surveillance domain. -In the Mind's Eye project [START_REF] Oltramari | Using Ontologies in a Cognitive-Grounded System: Automatic Action Recognition in Video Surveillance. 8. OWL Web Ontology Language Overview[END_REF] they used machine learning algorithms for features extraction from a camera input step, and to know patterns and detect suspicious behaviours they match them using a cognitive model operating over those visual features. Then the output is filtered back in the form of new knowledge patterns into the cognitive model and as feature utility into the perceptual algorithms.

In [START_REF] Ly | Specific Behavior Recognition Based on Behavior Ontology[END_REF] a behaviour ontology is proposed. The ontology is evaluated in the PETS 2006 and PETS 2007 datasets. This approach use prior knowledge, for detecting behaviour without training data of entire process. They sets that a specific behaviour can be acted in various ways but they still share a general plot.

Another interesting ontologies can be found in [START_REF] Kazi Tani | Events Detection Using a Video-Surveillance Ontology and a Rule-Based Approach[END_REF], [START_REF] Xue | Ontology-based surveillance video archive and retrieval system[END_REF], [START_REF] Fernández | Interpretation of complex situations in a semantic-based surveillance framework[END_REF], [START_REF] Akdemir | An ontology based approach for activity recognition from video[END_REF], Etiseo project [START_REF] Nghiem | ETISEO, performance evaluation for video surveillance systems[END_REF], [START_REF] Bai | Video semantic content analysis based on ontology[END_REF], [START_REF] Francois | VERL: an ontology framework for representing and annotating video events[END_REF].

Table VIII-1: Comparison between the background subtraction, the temporal differencing, the optical flow approaches, block matching and feature correspondence.

Approach

VIII.3. Appendix 3: Argumentation of the maximization equation

In this study, as shown in the sub-section III.3.5, the problem is not object detection or face localization, but it is a classification. Then, for the probabilistic approach, in the maximization equation, Arg max 𝑁,𝑁 2 𝑃[(𝑋 ≥ 𝑁 2 ) ∩ (𝑌 < 𝑁 2 )], X and Y are not completely independent, in addition, Y=N-X. For that, it seems enough to maximize only the X term without the Y term (arg max 𝑁,𝑁 2 𝑃[(𝑋 ≥ 𝑁 2 ) ). But if we use, for the maximization equation (arg max 𝑁,𝑁 2 𝑃[(𝑋 ≥ 𝑁 2 ) ), the same graphical representation that was introduced as a numerical resolution for the maximization problem (arg max 𝑁,𝑁 2 𝑃[(𝑋 ≥ 𝑁 2 ) ∩ (𝑌 < 𝑁 2 )]), we will have the graphical representation showing

. Object detection

Mainly visual surveillance analysis approaches start, after detecting regions corresponding to moving objects in the frames and images, by locating objects of interest in the scene. Following that, object classification and tracking, also behaviour analysis and recognition are significantly dependent on it.

As mentioned, in visual surveillance, motion is the key feature, and the temporal information is widely exploited by detection approaches. The process of object detection usually starts with environment/background modelling [START_REF] Neves | Biometric recognition in surveillance scenarios: a survey[END_REF], [START_REF] Revathi | A SURVEY OF ACTIVITY RECOGNITION AND UNDERSTANDING THE BEHAVIOUR IN VIDEO SURVEILLANCE[END_REF], (W. [START_REF] Hu | A Survey on Visual Surveillance of Object Motion and Behaviors[END_REF]), (W. [START_REF] Hu | A Survey on Visual Surveillance of Object Motion and Behaviors[END_REF], and motion segmentation.

Several basic conventional approaches for object detection can be used: Background subtraction, temporal differencing, optical flow and feature-based approach; the reader is referred to chapter III and Appendix II for more explanation, references and comparison for each of these approaches.

Detecting objects in still images in the late years showed very good results. Different object types were detected using deep CNNs [START_REF] Szegedy | Going Deeper With Convolutions[END_REF], [START_REF] Girshick | Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[END_REF], [START_REF] Girshick | Fast R-CNN[END_REF], [START_REF] He | Deep Residual Learning for Image Recognition[END_REF], [START_REF] Redmon | You Only Look Once: Unified, Real-Time Object Detection[END_REF], (K. [START_REF] Wang | Dictionary Pair Classifier Driven Convolutional Neural Networks for Object Detection[END_REF]. He et al. [START_REF] He | Deep Residual Learning for Image Recognition[END_REF], and ResNet152 classifier CNN [START_REF] He | Deep Residual Learning for Image Recognition[END_REF], proposed a novel Residual Neural Network (ResNet) which train very deep networks with over one hundred layers, resulting in a very good performance classifying 1000 object classes on a public image dataset. You Only Look Once YOLO [START_REF] Redmon | You Only Look Once: Unified, Real-Time Object Detection[END_REF], YOLOv2 [START_REF] Redmon | YOLO9000: Better, Faster, Stronger[END_REF] and Single Shot MultiBox Detector SSD [START_REF] Liu | SSD: Single Shot MultiBox Detector[END_REF], Deconvolutional Single Shot Detector (DSSD) [START_REF] Fu | DSSD : Deconvolutional Single Shot Detector[END_REF] generated multiple boxes from the image, then simultaneously aim to predict these bounding boxes for each object on image and correspondent class labels for them, and apply classification according to probabilistic scores. NoScope (D. [START_REF] Kang | NoScope: Optimizing Neural Network Queries over Video at Scale[END_REF] has improved the filtering of frames, then they perform heavy CNNs. A Faster R-CNN is proposed by [START_REF] Ren | Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[END_REF], to share full-image convolutional features with the detection network, they uses the fully convolutional network called Region Proposal Network (RPN) which, mainly, predicts object boundaries and scores at each position. The RPN is trained to produce high-quality region proposals, to be used by Fast R-CNN for detection. Then, counting on the "attention" mechanisms, they merge RPN and Fast R-CNN into a single network by sharing their convolutional features.

For the object detection in videos, also many works were introduced. Han et al. [START_REF] Han | Seq-NMS for Video Object Detection[END_REF] proposed a NMS method to sequences still-image detections, then apply the sequence-level NMS on the results. Weaker class scores are then boosted by the detection on the same sequence. [START_REF] Kang | Object Detection in Videos With Tubelet Proposal Networks[END_REF] proposed a T-CNN Tubelets with Convolutional Neural, the proposed network generates many of tubelet proposals simultaneously for object detection from videos. [START_REF] Galteri | Spatio-Temporal Closed-Loop Object Detection[END_REF] in the goal of improving window ranking, they feed back their algorithm with the object detection results on previous frame, using a closed loop framework.

VIII.6. Appendix 6: Features extraction

Five types of features were extracted, spatial, temporal, inter-objects, inter-frames and trajectory features; here we present detailed explanations on the first three types as follow:

A. Object spatial Features: after the segmentation, in each of the scene frames, the following features, shown in Table VIII-13

Table VIII-13: Spatial features of objects

Obj_width(obj)

The object width normalized by the frame width (bbox_width*100/frame_width).

Obj_height(obj)

The object height normalized by the frame height (bbox_height*100/frame_height).

xC_pos(obj)

The x position of the object centroid C(obj) with respect of the screen (x*100/ frame_width).

yC_pos(obj)

The y position of the object centroid C(obj) with respect of the screen (y*100/frame_height).

S_bbox(obj)

The bbox surface normalized by the frame surface (bbox_width * bbox_height *100/ frame_width * frame_height).

S1_obj (obj)

The object surface normalized by the frame surface (nb_pixels(obj)*100/ frame_width * frame_height).

S2_obj (obj)

The object surface normalized by the bbox surface (nb_pixels(obj)*100/ bbox_width * bbox_height).

Bmask(obj)

The bbox binary mask differentiating the object pixels from background (see Figure VIII-9).

Intensity(obj)

The bbox grey matrix presenting the object pixels.

RGB(obj)

The bbox RGB colour matrix presenting the object pixels (see Figure VIII-9).

Int_mean(obj)

The mean of the grey matrix of the object pixels.

RGB_mean(obj)

The mean of the RGB colour 3D matrix of the object pixels.

Int_std(obj)

The standard deviation (std) of the grey matrix of the object pixels.

RGB_std(obj)

The standard deviation (std) of the RGB colour 3D matrix of the object pixels.

P_obj (obj)

The object perimeter normalized by the object surface (p(obj)*100/ nb_pixels(obj)).

Hu(obj)

The Hu invariant moments vector (7 moments)[Invariant Moments] of the object.

Hu_comb(obj)

The 7 Hu invariant moments of the object combined in one value.

Reliability(obj)

The reliability of the segmented object in the frame, it depends on which is the object predicted (occluded), directly detected after appearance and before disappearance.

Type_obj(obj)

The object type if deformable or not, according to the algorithm of the chapter III. The bbox surface change normalized by the frame surface (S_bbox f (obj)-S_bbox f-1 (obj)).

S1_obj_change (obj)

The object surface change normalized by the frame surface (S1_obj f (obj)-S1_obj

f-1 (obj)).

S2_obj_change (obj)

The object surface change normalized by the object surface ((S1_obj f (obj)-S1_obj

f-1 (obj))*100/ S1_obj f-1 (obj)).

P_obj_change (obj)

The object perimeter change normalized by the object surface (P_obj f (obj)-P_obj f-1 (obj)).

Dist1(obj)

The distance between the object centroids (current frame f and f-1) normalized by the frame surface (d(C f (obj), C

f-1 (obj))*100/ frame_width*frame_height).

Dist2(obj)

The distance between the object centroids (current frame f and f-5) normalized by the frame surface, (d(C f (obj), C f-5

(obj))*100/ frame_width*frame_height).

Speed(obj)

The speed between the object centroids (current frame f and f-5) normalized by the frame surface, as 1 sec =25 frames, (Dist2(obj)/(5/25)).

Angle1(obj)

The angle formed by 3 object centroids (f-1, current frame f and f+1).

Angle2(obj)

The angle formed by 3 object centroids (f-5, current frame f and f+5).

Hu_changes(obj)

The Hu invariant moments vector changes (7 moments) of the object (sum(abs(Hu f (obj)-Hu f-1 (obj))).

Huα_changes(obj)

The Hu invariant moments α changes of the object (Huα

Form_change1(obj)

The change of the object forms normalized by the object surface. This feature calculates the difference of surfaces between binary masks, after co-centring the centroids, by counting the number of pixels which do not coincide, divided by the nb_pixels(obj), (see Figure .

Form_change2(obj)

The upper part only of the change of the object forms normalized by the object surface. We took the upper part only because it is more reliable than the lower part, because of the shadow. This feature calculates upper part of the difference of surfaces between binary masks, after co-centring the centroids, by counting the number of pixels above the centroid which do not coincide, divided by the nb_pixels(obj), (see Figure .

Obj_width_change(obj)

The normalized change of the object width ((Obj_width