Keywords: Augmented Reality xviii Application Specific Integrated Circuit 74 AVC, Advenced Video Coding xix AXI, Advanced eXtensible Interface 75 BD-BR, Bjontegaard Delta bitrate 41, 42 Context-Adaptive Binary Arithmetic Coding 9 Configurable Logic Blocks 77 53 DCT, Discrete Cosine Transform 9

V

ideo contents visualization has been revolutionized over the last decade with the advent of video-on-demand services, web-TV, video-sharing sites, live streaming service for individuals, and broadcast platforms offered by social networks. This led to an explosion of internet traffic. According to a recent Cisco study, video-driven internet traffic will quadruple between 2017 and 2022 and will represent 82% of overall internet traffic. The appearance of new video content, such as 360°video, Virtual Reality (VR), High Frame Rate (HFR) and the advent of very high spatial resolution 8K or even 16K leads to a significant increase in the amount of data to be transmitted. Consequently, efficient compression is essential to store or transmit this huge amount of data. Despite the considerable performance achieved by the video coding standards, the existing compression techniques showed their limitations and it is becoming increasingly difficult to meet the growing demands of data. Therefore, the adoption of new approaches such as machine learning based methods has great potential to address this challenge and can provide very promising results. The objective of this thesis is to introduce advanced techniques to significantly reduce the complexity of the High Efficiency Video Coding (HEVC) and the Versatile Video Coding (VVC) standards, while preserving the bitrate gain and ensuring a better video quality for users. These techniques, based on machine learning provide better performance in classification, in prediction and in efficient compression vs classical algorithms.

First, I would like to witness my gratitude to Almighty God, who gave me the courage and strength to pursue this thesis and opened the gates of knowledge for me.

I have started my final internship project for my master's degree in Electronics and

Microelectronics Lab Research EµE. It was March 2017. I never thought this training would be the reason for choosing me to pursue a long and fruitful Ph.D. journey full of fluctuations (The 1 st registration was on January 23, 2019). What's for sure is that I will never forget this opportunity which was wonderful, overwhelming, and full of lessons and responsibilities. Thankfully, my Ph.D. was completed successfully in which we delivered many journals and international conferences. This wouldn't have happened without the help and support of countless people over the past three years.

First of all, I am deeply grateful to my supervisors Mrs. Fatma Ezahra SAYADI

and Mrs. Randa KHEMIRI who gave me this opportunity and believed in me from the very beginning. Knowing that this Ph.D. experience wasn't always easy, it was always a pleasure, with a lot of excitement, to be up to the challenge and to beat paper deadlines.

Working with you improved me a lot as a student, as a researcher, and as a person. It wasn't straightforward for me to understand how to think like a researcher. You have taught me, both consciously and unconsciously, how good work is done. I appreciate all your contributions of time and ideas to make my Ph.D. experience productive. You have always listened to my ideas and discussions with you frequently led to progress. Your ability to approach research problems and your high scientific standards set an example. I admire your ability to balance research interests and personal pursuits. I am thankful for the excellent example you have provided me as a successful and ambitious researcher. I would first of all like to thank the members of the jury for their presence, for their careful reading of my thesis as well as for the remarks they will address to me during ii this defense in order to improve my work. I thank Professors Ali DOUIK and Khaled BEN KHALIFA for finding the time to read my thesis and for their valuable feedback.

I would like to thank Professor Chokri SOUANI for being examinator of this jury.

And finally, I would like to thank Professor Nejib HASSEN for the honor he gave me when he agreed to chair this jury. I would like to thank all EµE laboratory members, especially the Lab head Pr. Mohsen MACHHOUT and Pr. Mohamed ATRI for their advice and their human qualities of listening and understanding. This work would never be completed without the support of my family. Words cannot describe my gratitude for my parents, my brothers, and my sisters. Without them, I could never have reached this current level of success. To all of you, thank you for your continuous encouragements and devotion. I would like to give special thanks to my friend Dr. Seifeddine MESSAOUD who have always encouraged me.

Soulef BOUAAFIA

iii """"""' Résumé L a visualisation de contenus vidéo a été révolutionnée au cours de la dernière décennie avec l'apparition des services de vidéo à la demande, de web-TV, de sites de partage de vidéos, de service de diffusion en direct pour les particuliers, et des plateformes de diffusion offertes par les reseaux sociaux. Ceci a conduit à une explosion du trafic internet. Selon une étude récente de Cisco, le trafic internet lié à la vidéo quadruplera entre 2017 et 2022 et représentera 82% du trafic internet global. L'apparition de nouveaux contenus vidéo, tels que la vidéo 360°, la Réalité Virtuelle (VR), le High Frame Rate (HFR) et l'avènement de très grandes résolutions spatiale 8K voire 16K conduit à une augmentation significative de la quantité de données à transmettre. Par conséquent, une compression efficace est essentielle pour stocker ou transmettre cette énorme quantité de données. Malgré les performances considérables obtenues par les normes de codage vidéo, les techniques de compression existantes ont montré leurs limites et il devient de plus en plus difficile de répondre aux demandes croissantes de données. Par conséquent, l'adoption de nouvelles approches telles que les méthodes d'apprentissage automatique représente un grand potentiel pour relever ce défi et peut fournir des résultats très prometteurs. L'objectif de cette thèse est d'introduire des techniques avancées pour réduire significativement la complexité des normes de codage vidéo High Efficiency Video Coding (HEVC) et Versatile Video Coding (VVC) tout en préservant le gain en débit et assurant une meilleure qualité de vidéo aux utilisateurs. Ces techniques basées sur l'apprentissage automatique offrent de meilleures performances en classification, en prédiction et en efficacité de compression par rapport aux algorithmes classiques.

List of Figures

W

ith the development of multimedia computing, communication and display technologies, many video applications have emerged, such as TV broadcasting, video-on-demand, video conference, mobile video, video surveillance, 3D videos and Augmented Reality (AR), which can provide immersive telepresence and realistic visual perception experience. These video applications have been widely employed for multiple roles in human daily life, such as manufacturing, communication, national security, military, education, medicine, and entertainment. Nowadays, video data has been the majority data traffic over the internet and its volume grows explosively each year.

The latest Cisco Visual Networking Index reports that Internet Protocol video traffic accounted for 75% of all Internet traffic in 2017, and they expect it to rise up to 82% by the year 2022 [START_REF] Cicero | Cisco predicts more ip traffic in the next five years than in the history of the internet[END_REF]. On that occasion, million minutes of video contents will be delivered through the network in every second. To further enhance the immersive and realistic visual experiences, more high-end video applications emerge, such as High and Ultra-High Definition content (HD, UHD), Virtual Reality (VR), High Frame Rate (HFR) and 360°video and the advent of very high spatial resolution 8K or even 16K, which require larger data volume to represent higher fidelity and more details. Meanwhile, the number of video clients and cameras in use grows rapidly as the video demands keep boost in recent years, as HDTV, surveillance cameras, laptop and smart phones.

The total amount of global video data doubles every two years, which is the bottleneck for data processing, storage and transmission.

General Introduction

Video coding is one of the basic technologies in video applications that allows video data to be structured and compressed more efficiently for computation, transmission and storage. It has been developed over three decades with four generations and the coding efficiency doubles every ten years. But there is a big gap as compared with the rapid growth of global video data doubling every two years. Achieving much higher compression efficiency and narrowing the gap in an effective way become urgent missions for video coding. Machine learning is a field of study that can learn from data, discover hidden patterns and make data-driven decisions. Due to its superior performance in learning from data, many emerging works have applied machine learning algorithms to video coding to further promote the coding performances, which becomes one of the most promising directions in both academic and industrial communities.

In this context, the advent of the video coding standard, High Efficiency Video Coding (HEVC), standardized in January 2013 [START_REF] Gary J Sullivan | Overview of the high efficiency video coding (hevc) standard[END_REF] has made it possible to broadcast the UHD content on the communication network. The HEVC provides nearly 50% bitrate gain in comparison to the H.264/AVC standard for the same quality. However, HEVC is still not efficient enough to endure the burden of video transmission and storage for various large popular applications based on 8K and 360°videos. For this reason, Versatile Video Coding (VVC) appears the most recent video coding standard developed jointly by JVET, as known H.266 [BCO + 21]. It is based on the same hybrid video coding block, as its predecessors from MPEG-2 to HEVC. VVC is designed to be both efficient and versatile to address today's media needs. This includes approximately 30% -50% bitrate reduction over HEVC [WHB + 20], as well as versatility by efficient coding of a wide range of video content and applications.

The main objective of this thesis is to significantly improve the coding efficiency of HEVC and VVC standards based on fast machine learning algorithms. This manuscript is structured in four chapters:

Chapter I : Video Coding and Artificial Intelligence Backgrounds

General Introduction

The first chapter introduces the most emerging video technologies nowadays. For this purpose, the hybrid aspects of the video coding standards are discussed first and then some essential modules to build a codec with this structure are detailed. In order to emphasize on the similarity of general structure between different video coding standards, some sections also provide equivalent historical and descriptive information from HEVC along with VVC. Meanwhile, we summarize the most challenges in video coding standards. After that, we introduce the recent advancements in machine learning and deep learning models and their categories. Finally, some related research based on video coding techniques are reviewed. Finally, the achieved results are discussed and a comparative study is made.

Chapter II : Machine

Finally, the last part of this thesis will be reserved for a general conclusion that summarizes the results found and lists the different perspectives.

Chapter I

Video Coding and Artificial Intelligence

I.1 Introduction

I

n this chapter, a brief review of the video compression structure is presented. For this purpose, the hybrid video coding standards scheme is first discussed and then some essential modules to build a codec with this structure are detailed. In order to emphasize on the similarity of general structure between different video coding standards, some sections also provide equivalent historical information from High Efficiency Video Coding (HEVC) along with Versatile Video Coding (VVC). Meanwhile, we summarize the most challenges in video coding standards. After that, we introduce the recent advancements in machine learning and deep learning models and their categories. Finally, this chapter provides a detailed literature review on different advanced video coding approaches that have been proposed.

The remainder of this chapter is organized as follows. Section I.2 presents the video compression history. The HEVC standard structure is described in Section I.3. Section I.4 introduces the VVC coding tools. Then, Section I.5 provides the video coding challenges. Afterwards, a detailed overview of artificial intelligence technique is exposed in Section I.6. The related research of video coding approaches is presented in Section I.7.

Finally, Section I.8 concludes this chapter.

I.2 Video Compression History

Every second in year 2021, more than a million minutes of video content will cross the network. It would take a person more than 5 million years to watch all videos of one month [START_REF] Cicero | Cisco predicts more ip traffic in the next five years than in the history of the internet[END_REF]. This forecast is convincing for video coding experts to think of more efficient compression tools and technologies. These technologies are expected to address various emerging video formats, namely High Dynamic Range (HDR), High Frame Rate (HFR), high resolution videos (e.g. 4K, 8K and beyond), immersive 360°videos, screen content and more. Video coding standards aim at bringing format compatibility between devices. This enables the playback of any video file conforming the syntax of a given standard, with any device supporting it. From the industrial point of view, such unity facilitates the interaction between all components of the broadcast chain, including consumer electronics manufactures, broadcasters, content providers etc. This convenience in interaction, if achieved, can significantly accelerate the progress of the broadcast industry as a whole.

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS

Most successful standardization acts of the MPEG were accomplished after its collaboration with the ITUT, in the late 90's. This joint collaboration, initially called Joint Video Team (JVT), then Joint Collaboration Team on Video Coding (JCT-VC), resulted in some of the most successful video coding standards in the family of "H.26x", notably H.264/Advanced Video Coding (AVC), developed in May 2003 [START_REF] Chen | Introduction to h. 264 advanced video coding[END_REF], and H.265/High Efficiency Video Coding (HEVC), finalized in January 2013 [START_REF] Gary J Sullivan | Overview of the high efficiency video coding (hevc) standard[END_REF]. In October 2015, another collaboration between MPEG and VCEG formed the Joint Video Exploration Team (JVET) [START_REF] Amir | Jvet encoder complexity analysis[END_REF] that was tasked with assessing the available compression technologies and exploring the requirements for a next-generation video compression standard. Hence, the new video coding standard called H.266/Versatile Video Coding (VVC) was standardized in July 2020 [BCO + 21]. After the history presentation of the different video coding standards, the two latest HEVC and VVC will be detailed in next sections, since they will be used in this thesis.

I.3 HEVC Standard

High Efficiency Video Coding (HEVC) is the sophisticated video coding standard, also known as H.265, standardized in 2013 by the JCT-VC [START_REF] Richardson | An introduction to high efficiency video coding[END_REF]. HEVC saves approximately 50% of bitrate for the same subjective video quality, with respect to its predecessor H.264/AVC standard. Thus, the HEVC codec is expected to ease the burden on global networks where High Definition (HD) and Ultra High Definition (UHD) video content is becoming more and more popular. HEVC is based on the basic hybrid structure as employed by previous standards since H.261. However, the standard contains series of incremental improvements [START_REF] Richardson | An introduction to high efficiency video coding[END_REF] In accordance, a typical video encoder compliant with the HEVC standard would start by dividing each frame into block-shaped regions, with the exact block partitioning being conveyed to the decoder. The first picture of the video sequence is coded using only intra picture prediction, i.e., the prediction of the blocks in the picture is only BACKGROUNDS From Figure I.2, the result from the prediction is subtracted from the original block and the residual information is then transformed by a linear spatial transform. The transform coefficients are then scaled, quantized, compressed and transmitted in the receiver, together with the prediction information. The encoder also integrates the processing loop of the decoder in order to generate the same pictures as the output of the decoder. These pictures are then stored in a decoded picture buffer, and will be used for the prediction of the subsequent pictures. In the following, the general features of the hybrid video coding scheme used in HEVC will be described with more details.

I.3.1 Sampled Representation of Pictures

Video sequence is typically captured using the RGB color space, which is not a particularity efficient representation for video coding. On the contrary, HEVC uses a more video coding friendly color space, the YCbCr, which divides the color space in 3 components: Y, known as luma, representing brightness; Cb and Cr, also known as chroma, which represent how much color deviates from gray towards blue and red, respectively.

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS

As the human visual system is more sensitive to brightness, the typically used sampling scheme follows the 4:2:0 structure, meaning that four luma components are sampled for every chroma component. HEVC also supports each sample pixel value with 8 or 10 bits precision, with 8 bits being the most commonly used for HEVC standard [B + 13] and 10 bits used for VVC standard [FJK + 20].

I.3.2 Block Partitioning in the HEVC Standard

In the former video coding standard H.264/AVC, Variable Macro-Block (MB) sizes ranging from 4×4 to 16×16 are supported [START_REF] Chen | Introduction to h. 264 advanced video coding[END_REF]. Whereas larger block sizes, reached at 64×64, are used in HEVC standard to facilitate the high definition video compression.

Additionally, more flexible partitioning of video frames is supported to improve the (CTU), is splitted into CUs using a quad-tree partitioning structure, and a CU can be further sub-divided into Prediction Units (PU) for inter-frame or intra-frame prediction and its transformation is performed using one or more Transform Units (TU).

I.3.3 Intra Prediction

In intra picture prediction, the information of adjacent CTU from the same picture is used for spatial prediction, as shown in Figure I.4. There are a total of 35 intra picture prediction modes available in HEVC, corresponding to 33 different directional modes, a DC and a planar mode. For directional mode encoding, the spatially neighboring decoded blocks are used as reference for the prediction, using the selected angle to cover the current PU. This mode is the most used for regions with strong directional edges.

Directional mode prediction is consistent across all block sizes and prediction directions.

DC mode encoding simply uses a single value matching the mean value of boundary samples for the prediction. Finally, the planar mode assumes an amplitude surface with a horizontal and a vertical slope derived from the boundaries. This mode is supported for all block sizes in HEVC.

I.3.4 Inter Prediction

In order to exploit the redundancies in the temporal adjacent images, inter-picture prediction based on previously coded pictures is an essential technique to obtain high compression rates. It consists of the application of the following two techniques: motion compensation and motion estimation. By using these techniques, pictures are predicted from previously encoded frames (uni-directional) or from previous and future frames (bi-directional), as shown in Figure I.5. The use of the bidirectional prediction is more complex, since it requires the video frames to be coded and stored out of order, so that future frames may be available. Before the application of motion compensation technique, the encoder has to find a block similar to the one it is encoding on a previous/future encoded frame, referred to as a reference frame. Such searching procedure is known as motion estimation, resulting in the identification of a motion vector, which points to the position of the best prediction block in the reference frame. However, since the identified block will most likely not be an exact match of the encoding block, the resulting difference (residue) has to be encoded and transmitted to the decoding end, so that it can be read by the decoder.

These residuals, originated from the difference between the predicted block and the actual block, are known as prediction errors.

The actual position of the prediction in the neighboring frames may be out of the sampling grid (where the intensity is unknown), so the intensities of the positions in between the integer pixels must be interpolated and the resolution of the motion vector

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS

increased accordingly. For the interpolation in fractional luma sample positions, an 8-tap filter is used, while a 4-tap filter is used for chroma samples.

I.3.5 Transform and Quantization

After the motion estimation, all the prediction error residuals are transformed into a set of coefficients for efficient transmission and storage. In the HEVC standard, as indicated in

I.3.6 Entropy Coding

In the HEVC standard a bitstream is produced using motion parameters, prediction modes, quadtree partitioning information, quantized transform coefficients and some other control data through entropy coding. Only one entropy coding method, Context-Adaptive Binary Arithmetic Coding (CABAC), is specified in the standard. Although there is no change made on the core algorithm of CABAC, it is optimized on the aspects of context modeling, adaptive coefficient scanning, coefficient coding, sign data hiding and so on to improve its throughput.

I.3.7 In-Loop Filters

Before writing the samples in the decoded picture buffer, they are processed first by a deblocking filter (DBF) and then by a sample adaptive offset filter (SAO). Block based coding schemes tend to produce blocking artifacts due to the fact that inner blocks are coded with more accuracy than outer blocks. To mitigate such artifacts, the decoded

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS

samples are filtered by a DBF. After the deblocking has been processed, the samples are processed by SAO, a filter designed to allow for better reconstruction of the original signal amplitudes, reducing banding and ringing artifacts. SAO is performed on a per CTU basis and may or may not be applied, depending on the filtering type selected.

I.4 VVC Standard

Versatile Video Coding (VVC) [BCO + 21] [HBA + 21] is the new generation video coding developed in July 2020, by the Joint Video Experts Team (JVET), as a successor of the HEVC [START_REF] Gary J Sullivan | Overview of the high efficiency video coding (hevc) standard[END_REF]. As the next standard for sophisticated video coding technology, VVC allows up to 30% -50% for bitrate savings while maintaining the same quality as HEVC. VVC has been designed to achieve improved compression capacity over previous standards such as HEVC, and at the same time to be highly versatile for effective use in a broadened range of applications. Some key application areas for the use of VVC particularly include UHD video (e.g. 4K or 8K resolution), video with a high dynamic range, and video for immersive media applications such as 360°omnidirectional video, in addition to the applications that have commonly been addressed by prior video coding standards.

Similar to its predecessor HEVC, VVC uses a block-based hybrid coding architecture with some coding tools that may be included or removed. The VVC architecture includes the inter-picture, intra-picture prediction, and transform coding with entropy coding.

I.4.1 Block Partitioning in the VVC Standard

In VVC, each picture is split into non-overlapping squares called CTUs. The largest CTU size allowed in VVC is 128 × 128 pixels, larger than the maximum size allowed in HEVC, 64 × 64. Large blocks improve the efficiency of coding flat areas such as backgrounds, especially for high-resolution videos such as HD and 4K. In order to efficiently represent highly detailed areas such as textures and edges, VVC employs a flexible partitioning scheme that can CTUs partition sized of 128 × 128 down CUs as small as 4 × 4 pixels. or ternary splits. The first is the quadtree split that is also available in HEVC, which can recursively split CTU into squared CUs down to 4 × 4 pixels, smaller than the 8 × 8 minimum CU size in HEVC. The second part consists of binary-tree and ternary-tree splits that partition a block into two and three rectangles respectively. Both binary and ternary tree splits can operate in either horizontal or vertical directions, be recursively applied and mixed together in a nested multi-type tree.

The block partitioning in VVC is highly flexible and provides about 8 percent bitrate reduction over HEVC. However, this flexibility comes at a computational cost, especially on the encoder side, where many more permutations need to be evaluated to select the optimal partition.

I.4.2 Intra Prediction

The number of directional intra modes in VVC is extended from 33, as used in HEVC, to

I.4.3 Inter Prediction

The basic concepts of uni-directional and bi-directional motion compensation from one or two reference pictures are mostly unchanged. However, there are some new tools

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS

that have not been used in the last video coding standard. For each inter-predicted CU, motion parameters consisting of motion vectors, reference picture indices and reference picture list usage index, and additional information needed for the new coding feature of VVC to be used for inter-predicted sample generation. The motion parameter can be signalled in an explicit or implicit manner. When a CU is coded with skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector delta or reference picture index. A merge mode is specified whereby the motion parameters for the current CU are obtained from neighbouring CUs, including spatial and temporal candidates, and additional schedules introduced in VVC. The merge mode can be applied to any inter-predicted CU, not only for skip mode. The alternative to merge mode is the explicit transmission of motion parameters, where motion vector, corresponding reference picture index for each reference picture list and reference picture list usage flag and other needed information are signalled explicitly per each CU.

Beyond the inter coding features in HEVC, VVC includes a number of new and refined inter prediction coding tools listed as follows; Extended merge prediction, 1/16th luma sample MV storage and 8×8 motion field compression, Bi-prediction with CU-level weight (BCW), and Bi-directional optical flow (BDOF), etc.

I.4.4 Transform and Quantization

The size of transform block is increased from 4 × 4 to 64 × 64 in the VVC standard compared to the HEVC standard. In addition to the DCT-II used in HEVC, a multiple transformation selection (MTS) scheme is also used for residual coding of intra and inter

I.4.5 Entropy Coding

In VVC, the CABAC technique is improved in comparison to the HEVC design. The three main modifications are: modified context modeling for transform coefficients, multi-hypothesis probability estimation with context-dependent updating speed and

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS

adaptive initialization for context models (e.g. initial probability states of context models for inter coded slices can be initialized by copying states from previously coded pictures).

Table I.1: Coding Tools of VVC vs HEVC

I.4.6 In-Loop Filters

In VVC, a remapping operation and three in-loop filters can be applied sequentially to the reconstructed picture to modify its representation domain and alleviate different types of artifacts. First, a new sample-based process called LMCS (Luma Mapping with Chroma Scaling) is performed. Then, a DBF is used to reduce blocking artifacts.

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS

SAO is then applied to the deblocked picture to attenuate ringing and banding artifacts.

Finally, an Adaptive Loop Filter (ALF) reduces other potential distortion introduced by the quantization and transform processes. The deblocking filter design is based on the one in HEVC but is extended with longer deblocking filters and a luma-adaptive filtering mode designed specifically for HDR video. While SAO is the same as in HEVC, and the deblocking is very similar, LMCS and ALF are new compared with previous standards.

The design of ALF in VVC consists of two operations: ALF with block-based filter adaption for both luma and chroma samples and a cross-component ALF (CC-ALF) for chroma samples.

I.5 Video Coding Challenges

During the last decade, multimedia services and video applications have significantly increased due to the huge progress in digital technologies. The emerging video applications and image representation offer an immersive and more natural viewing experience.

However, these new services require both higher quality and resolution (4K, 8K) to satisfy the quality of service required by the end users. To meet the increasing demands for video content at better qualities and higher resolutions, video compression technology is being researched and developed, due to its higher performance. However, this unmatched performance is achieved by increasing the encoder computational complexity mainly due to its block partition structure. Indeed, the complexity reduction has always been a popular challenge in the video coding field. For example, Figure I.9 shows that the greatest complexity lies in the selection of the optimal prediction mode, especially in the inter-mode [START_REF] Gabriel Cebrian-Marquez | Adaptive inter cu partitioning based on a look-ahead stage for hevc[END_REF].

In this context, many researchers aim to reduce the complexity for each standard

I.6 Artificial Intelligence: New Advancements and Innovations

Artificial Intelligence (AI) is a branch of computer science that deals with simulation of human intelligence by machines processes and computational rationality. The term may also be applied to any machine that exhibits traits associated with a human mind such as learning and problem-solving. AI is a computer system able to perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision making, and translation between languages. Machine learning and deep learning are subsets of AI, which are described in the following sections.

I.6.1 Machine Learning

The learning activity is essential for the human beings in order to understand and recognize various parameters such as a voice, a person, an object, and others. Supervised learning systems make use of labeled datasets [START_REF] Sotiris B Kotsiantis | Supervised machine learning: A review of classification techniques[END_REF]. This training set of input-output pairs is used to find a deterministic function that maps any input to an output, predicting future input-output observations while minimizing errors as much as possible. While Unsupervised learning systems use unlabeled datasets to train the system [START_REF] Hastie | Unsupervised learning. the elements of statistical learning[END_REF]. The objective of unsupervised learning is to derive structure from unlabeled data by investigating the similarity between pairs of objects, and is usually associated with density estimation or data clustering. Reinforcement learning systems do not experience a fixed dataset, but a feedback loop between the system and its experiences [START_REF] Pack | Reinforcement learning: A survey[END_REF]. A dynamic environment is considered in which state-action-reward triples are observed as the data. The objective of reinforcement learning is mapping situations to actions with the goal of maximizing rewards. Other existing learning systems that are a combination of two categories, such as semi-supervised learning that uses both labeled and unlabeled data [START_REF] Chapelle | Semisupervised learning[END_REF].

Here, we limit our focus to supervised learning algorithms. There are a wide variety of tasks exist that could be solved with machine learning. However, two popular machine learning tasks are regression analysis and classification. Commonly used algorithms for classification technique [START_REF] Sotiris B Kotsiantis | Supervised machine learning: A review of classification techniques[END_REF] include k-Nearest Neighbor, Support Vector Machine, Naïve Bayes, and Decision Trees, etc...In the following, we focus on describing the Support Vector Machine (SVM) considered as the most useful algorithm, due to its

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS

ability to solve classification task problems. Indeed, SVM is a class of learning algorithm, initially used for discrimination that is, predicting a binary qualitative variable which is then generalized forecast a quantitative variable. In case of discriminating a dichotomous variable, they are based on the search for the optimal margin hyperplane.

I.6.2 Deep Learning

The past decade has witnessed the emerging and booming of Deep Learning (DL), a class of techniques that are increasingly adopted in the hope of approaching the ultimate goal of artificial intelligence [START_REF] Arif Wani | Advances in deep learning[END_REF]. DL belongs to machine learning technology, and has the distinction of its computational models, known as deep artificial neural networks or deep networks for short, which are composed of multiple (usually more than three) processing layers, each layer is further composed of multiple simple but non-linear basic computational units. One benefit of such deep networks is believed to be the capacity for processing data with multiple levels of abstraction, and converting data into different kinds of representations. Note that these representations are not manually designed; instead, the deep network including the processing layers is learned from massive data using a general machine learning procedure. DL eliminates the necessity of handcrafted representations, and thus is regarded useful especially for processing natively unstructured data, such as acoustic and visual signal, since the processing of such data is considered a long-standing problem in the field of artificial intelligence.

Specifically for processing image/video, DL using Convolutional Neural Network (CNN) has revolutionized the paradigm in computer vision and image processing [START_REF] Bouaafia | Fast cu partition-based machine learning approach for reducing hevc complexity[END_REF].

CNN is one of the most commonly used supervised deep learning models, which is described in this chapter. This network structure was first proposed by Fukushima in 1988 [START_REF] Fukushima | Neocognitron: A hierarchical neural network capable of visual pattern recognition[END_REF]. In the 1990s, LeCun et al.

I.6.2.1 Convoluional Layer

Convoluional Layer is the core building block of CNN network, in which its parameters consist of a set of learnable filters also known as kernels. The main task of the convolutional layer is to detect features found within local regions of the input image that are common throughout the dataset and mapping their appearance to a feature map.

A feature map is obtained for each filter in the layer by repeated application of the filter across sub-regions of the complete image, i.e., convolving the filter with the input image, adding a bias term, and then applying an activation function. Therefore, four important hyperparameters in the convolutional layer are used, such as Filter Size, Number of Filters, Stride, and Zero Padding. The following equation shows the convolution operation.

x l j = f   i∈M j x l-1 i × k l ij + b l j   (I.1)
where x l j is the output of the current layer, x l-1 i is the previous layer output, k l ij is the kernel for the present layer, and b l j are the biases for the current layer. M j represents a

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS

selection of input maps.

I.6.2.2 Pooling Layer

In CNN, the sequence of convolution layer and activation function layer is followed by an optional pooling or down-sampling (also sub-sampling) layer to reduce the spatial size of the input and thus reducing the number of parameters in the network. A pooling layer takes each feature map output from the convolutional layer and down-samples it, i.e., pooling layer summarizes a region of neurons in the convolution layer. Two types of operations are mostly performed in this layer: Average pooling or max-pooling. In the case of the average pooling approach, the function usually sums up over N×N patches of the feature maps from the previous layer and selects the average value. On the other hand, in the case of max-pooling, the highest value is selected from the N×N patches of the feature maps. The poling operation can be defined in equation I.2, where down(•) represents a sub-sampling function.

x l j = down x l-1 i (I.2)

I.6.2.3 Fully Connected Layer

At the end, the stack of convolutional and pooling layers act as feature extraction stage while as the classification stage is composed of one or more fully connected layers followed by an activation function layer. The process of convolutional and pooling continues until enough features are detected. Next step is to make a decision based on these detected features. In case of classification problem, the task uses the detected features in the spatial domain to obtain probabilities that these features represent each class, that is, obtain the class score. This is done by adding one or more fully connected layers at the end. In fully connected layer, each neuron from previous layer is connected to every neuron in the next layer and every value contributes in predicting how strongly a value matches a particular class.

Additionally, a fully connected layer is connected to all features, and it is prone to overfitting. Overfitting refers to the problem when a model is trained and it works so

I.6.2.4 Activation Functions

The output of each convolutional layer is fed to an activation function layer. The activation function layer consists of an activation function that takes the feature map produced by the convolutional layer and generates the activation map as its output. The activation function is used to transform the activation level of a neuron into an output signal.

There are many activation functions and some of the commonly used activation functions are as follows: Rectified Linear Unit (ReLU) has gained some importance in recent years and currently is the most popular activation function for deep neural networks.

Neural networks with ReLU train much faster than other activation functions. ReLU simply computes the activation by thresholding the input at zero. In other words, a rectified linear unit has output 0 if the input is less than 0, and raw output otherwise.

It is denoted as.

f (x) = max(0, x) (I.3)
The sigmoid function is mathematically represented in the equation I. the input layer and uses these errors to calculate the desired gradients. This description makes clear the incredible utility and computational efficiency of the backpropagation algorithm. We can calculate all the derivatives using a single "forward" and "backward" pass of the neural network, equations are summarized in A. This computational efficiency is crucial since we must calculate the gradient with respect to all parameters of the neural net at each step of gradient descent.

I.7 Related Research

The existing video coding complexity reduction works can be generally classified into two categories: heuristic and learning based approaches. This section reviews the complexity reduction approaches in these two categories.

I.7.1 Heuristic Methods

In heuristic methods, several fast decision algorithms have been introduced properties, temporal and spatial correlation, which limit their applicability and may be difficult to handle the situations with various contents, complex coding structures.

I.7.2 Learning Methods

The past few years have exhibited great success in applying machine learning tools to enhance the video coding. In this vein, great efforts have been carried out to integrate machine learning tools in order to predict the CU partition to reduce HEVC complex- Overall, the main target of video coding is to minimize the bitrate while maintaining the visual quality. There are three key requirements on the video coding [START_REF] Ohm | Vision, applications and requirements for high efficiency video coding (hevc)[END_REF], including high compression ratio, low complexity, and high visual quality. In this context, this thesis contribution proposes to integrate the advancement techniques in the video coding standards, in order to achieve a compression efficiency considerably higher than the old video compression technologies.

I.8 Conclusion

This chapter introduces the basic building blocks of a video compression system and a detailed description of the HEVC and VVC standards, which are the current state-ofthe-art video coding standards. The coding tools comparison of HEVC versus VVC is provided. Then, the chapter also presents a video coding standards challenges. Moreover, the recent advancements technologies, such as artificial intelligence, machine learning and deep learning and their tools are surveyed. Finally, the related researches on HEVC and VVC complexity reduction are reviewed.

In the next chapter, in order to overcome the HEVC complexity, we propose to integrate machine learning solutions in HEVC standard to predict the CU partition at inter-mode. This chapter will provide more details about the three proposed machine learning algorithms instead of traditional rate-distortion optimization search in HEVC standard.

II

RD cost _CU > 3 k=0 RD cost _sub_CU (k) (II.1)
The CU partition can be considered as a combination of binary classifiers {F l } 3 1 at three levels of decisions l ∈ 1, 2, 3 on whether to split a parent CU into sub-CUs. Accord-

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY

ing to the CTU, we assume that the CUs are denoted as CU , CU i , CU i,j corresponding to depth 0,1,2,3, where i, j ∈ 0, 1, 2, 3 are the index of sub-CUs. In each CU depth, we need to determine whether to split the current CU or not. The overall CU partition in a CTU is extremely complex, due to the large number of possible pattern combinations.

For example, for a 64 × 64 CU, if F 1 (CU )= 1, it will be split into four 32 × 32 CUs, i.e., {CU i } 3 i=0 . Since for each CU i there exist 1 + 2 4 = 17 splitting patterns in {CU i , j} 3 j=0 , the total number of splitting patterns for CU is 1 + 17 4 = 83522. There are too many types of CU partitions and it is hard to be solved by a single multi-class classification in one step. However, due to the large number of pattern combinations, the prediction is adopted at each decision level to yield F1 (CU ), { F2 (CU i )} 3 i=0 , and { F3 (CU i,j )} 3 i,j=0 , which denotes the predicted F 1 (CU ), {F 2 (CU i )} 3 i=0 , and {F 3 (CU i,j )} 3 i,j=0 , respectively.

II.3 Proposed CU Partition based on Machine Learning

II.3.1 CU Partition based on SVM

In machine learning theory, SVM is a supervised learning tool that performs classification analysis [START_REF] Cortes | Support-vector networks[END_REF]. In particular, the video coding mode decision process can be considered as a classification problem. A hyperplane technique is used in SVM to separate data from one space at one dimension to another at a larger dimension. SVM can transform data in a larger dimensional space by nonlinear transformation, if the data points are clearly not linearly separable in the input space. To separate the two classes of data points, SVM maps the sample data into a hyperspace. In addition, the main goal of SVM is to solve linear and nonlinear problems in order to find an optimal hyperplane. SVM classifier creates a hyperplane in order to maximize the margin between hyperplanes and support vectors [HCP + 18].

The CU split decision can be modeled as a binary classification problem, with classes split and non-split [START_REF] Bouaafia | Svm-based inter prediction mode decision for hevc[END_REF]. Here, we propose an online SVM as a machine learning technique, since it is robust and popular in solving the binary classification problem with significant computational advantages. The main idea is to find a hyperplane that can separate the training samples of different classes while maximizing the margin be-

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY

tween these classes in order to determine the CU splitting level. According to equation II.2, the ideal weight vector w is a linear combination of support vectors. Therefore, the support vectors are the training points that minimize the misclassification. Given training set with N samples, ({x i , y i } N i=1 ), x i ∈ R n , while y i ∈ ({-1, 1}), the hyperplane parameterized by the normal vector w that maximizes margins can be found by solving the optimization problem.

min w γ 2 w 2 + 1 n n i=0 max(0, 1 -y(w • x)) (II.2)
where γ ≥ 0 is the smoothing parameter and is defined by: γ=1/nC, where C is the parameter which need to be tuned during SVM training.

Mathematically, SVMs handle such situations by using a kernel function which maps the data to a different space where a linear hyperplane can be used to separate classes.

In this work, Gaussian Radial Basis Function (RBF) is applied as the kernel function, which is defined as:

K(x i , x j ) = exp(- x i -x j 2 2σ 2 ) (II.3)
Our approach therefore consists in determining when a 2N × 2N block has to be "Split"

or "Not-Split". The input feature vector is denoted by x i and y i is the output label indicating CU splitting or not. The following equation is the discriminant function:

f(x) = w T φ(x) + b (II.4)
where the normal vector is denoted by w. The function φ(x) maps feature vector x, and b is the bias.

The current CU splitting decision should be determined in each CU depth. Therefore, an SVM classifier is used in each CU depth to get the best combination of CU, PU and TU via evaluating the RD cost of all possible modes. The preprocessing layer are residual CUs of CU, CU i or CU i,j , corresponding to the three levels. Therefore, the residual block is subtracted by the mean intensity values to reduce the variation of the input CTU samples. Specifically, at the first level of CU partition, the mean value of CU is removed in accordance with the output of F1 (CU ).

At the second level, four CUs {CU i } 3 i=0 are subtracted by their corresponding mean values, matching the 2 × 2 output of ({ F2 (CU i )} 3 i=0 ). At the third level, {CU i,j } 3 i,j=0

remove the mean values in each CU for the 4 × 4 output ({ F3 (CU i,j )} 3 i,j=0 ).

After preprocessing layer, the three convolutional layers are used to extract features from data at all levels. The convolution layer is a mathematical operation that takes two inputs such as CU partition and filters. In each layer, the convolution kernels of all three levels have the same size. In our work, at the first convolutional layer, 16 kernels are used to extract the low features maps for the CU partition. At the second and third layers, feature maps are sequentially convoluted twice with 2×2 kernels (24 filters for the second layer and 32 filters for the third layer) to generate features at a higher level.The strides of all the above convolutions are equal to the widths of the corresponding kernels for non-overlap convolution.

The below design of the convolutional layer is in accordance with all possible nonoverlap CUs at different sizes for CTU partition. At the end of the convolution, through the concatenation layer, the final feature maps are concatenated together and then flatten into a vector. In the following fully connected layers, features generated from the whole CTU are all considered to predict the CU partition at each single level.

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY

( F1 (CU )) in 1 × 1, ({ F2 (CU i )} 3 i=0 ) in 2 × 2 and ({ F3 (CU i,j )} 3 i,j=0
) in 4 × 4 at three levels, respectively. In Deep CNN structure, the early termination may result in the calculation of the fully connected layers at levels 2 and 3 being skipped, thus saving computation time. Specifically, if CU is decided not to be split at level 1, the calculation of ({ F2 (CU i )} 3 i=0 ) is terminated early at level 2. If {CU i } 3 i=0 are all not split, the ({ F3 (CU i,j )} 3 i,j=0 ) at level 3 do not need to be computed for the early termination. The ReLU function is used to activate all convolutional layers and hidden fully connected layers, since this function has better convergence speed [START_REF] Glorot | Deep sparse rectifier neural networks[END_REF]. Moreover, since all the labels for splitting or non-splitting are binary, all the output layers in three levels are activated with the sigmoid function.

II.3.3 Training Phase

This section presents the training process for the proposed Deep CNN as shown in For learning our Deep CNN model, we assume that the cross entropy is applied as a loss function, which is defined in the equations II.5 and II.6:

L = 1 N N n L n (II.5)

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY

L n = Y (F n 1 (CU ), F n 1 (CU )) + i∈{0,1,2,3} Y (F n 2 (CU i ), F n 2 (CU i )) + i,j∈{0,1,2,3} Y (F n 3 (CU i,j ), F n 3 (CU i,j )) (II.6)
where Y denotes the cross entropy between the ground truth labels and the predicted labels. The labels predicted by our Deep CNN are represented by ({ F1 (CU )), ({ F2 (CU i )} 3 i=0 ), and ({ F3 (CU i,j )} 3 i,j=0 )} N n=0 .

We use the Tensorflow-GPU deep learning framework to train our proposed Deep CNN on an NVIDIA GeForce GTX 480 GPU that can dramatically improve speed during training compared to the CPU. We adopt a batch mode learning method with a batch size of 64 where the momentum of the stochastic gradient descent algorithm optimization is set to 0.9. To train our Deep CNN, the base learning rate is set to decay exponentially to 0.01, changing every 1, 000 iterations. The total number of iterations

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY

was 2, 000, 000. Finally, the trained model can be used to predict the CU partition at HEVC inter-mode. At level l, F C 1-l (t) represents the deep CNN input features at frame t, and

II.3.4 CU

F C l (t-1)
is the output features of the LSTM model of frame t-1. These three gates are presented in the following equations:

i l (t) = σ(W i • [F C 1-l (t), F C l (t -1)] + b i ) o l (t) = σ(W o • [F C 1-l (t), F C l (t -1)] + b o ) f l (t) = σ(W f • [F C 1-l (t), F C l (t -1)] + b f ) (II.7)
where the sigmoid function is denoted by σ(•).

c l (t) = i l (t) tanh(W c [F C 1-l (t), F C l (t -1)] + b c ) +f l (t) c l (t -1) (II.8)
where signifies the element-wise multiplication. The output of the LSTM cell F C l (t) can be determined as follows:

F C l (t) = o l (t) c l (t) (II.9)

II.3.5 Training Phase

In the training phase, the LSTM model was trained from the training set of the inter database given in Table II.1, which minimizes the loss function between the ground truth and the prediction of CTU partition.

L n (t) = Y (F n 1 (CU, t), F n 1 (CU, t)) + i∈{0,1,2,3} Y (F n 2 (CU i , t), F n 2 (CU i , t)) + i,j∈{0,1,2,3} Y (F n 3 (CU i,j , t), F n 3 (CU i,j , t))
(II.10) However, over N training samples alongside the T -f rames, the LSTM network can be trained by optimizing the cost function, as defined in equation II.11.

L = 1 N T N n T t L n (t) (II.11)
The training parameters were defined as follows; batch size, learning rate, and LSTM length (T ) were set to 64, 0.001, and 20, respectively. Finally, the trained model was saved to be used after (in the framework), which aims to predict the inter-coding CU partition. For the test, the LSTM model works in stages, that is, when the prediction of the CU partition at frame t -1 is complete, the state and the output of the frame t are computed. To further enhance the RD performance and reduce the inter-coding

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY

II.4.2 Performance Metrics

The

T= T P roposed -T Original T Original × 100 (%) (II.12)
where T P roposed and T Original are the coding time of the proposed approach and the original HEVC algorithm, respectively. For further performance evaluation of the proposed scheme, Table II.4 shows the coding performance between the proposed CNN-LSTM framework and the deep CNN [START_REF] Bouaafia | Fast cu partition-based machine learning approach for reducing hevc complexity[END_REF]. The proposed scheme CNN-LSTM is better than the deep CNN in terms of computational-complexity and RD performance. Specifically, the execution time of this method is 58.60% on average, exceeding the 53.99% obtained with deep CNN only. of 75% and 58.60% on average and gives an increase in BD-BR of 1.78% with a little reduction in BD-PSNR of -0.053 dB.

II.4.3 Performance Evaluation with Online SVM and Deep CNN

II.4.4 Performance Evaluation with Deep CNN and CNN-LSTM

In fact, the proposed approach achieves a higher computational-complexity reduction for video sequences with low motion activities and homogeneous regions, where the blocks CU partition is larger and the percentage of splitting cases is lower, such as "Kris-tenAndSara" video sequence. Similarly, the existing methods prove a high encoding time

for class E video sequences. For example, [LZZ + 19] achieves 64% encoding complexity and 1.58% BD-BR increase for sequence "KristenAndSara", as shown in Table II.5. For the same sequence, [START_REF] Tahir | Fast video encoding based on random forests[END_REF] gives 77% time saving with an increase in the BD-BR of 3.30% on average of four QPs. In addition, the work proposed in [XLW + 18] achieves 67.23% encoding time with 1.55% BD-BR on average. With regard to the ultra-high definition sequences like "PeopleOnStreet", the computational complexity reduction of our proposed approach is slightly lower, since these sequences have high motion and camera movement, which are encoded in a small CU partition. Hence, the proposed scheme performs better in terms of both RD performance and complexity reduction of HEVC as compared to the previous works. Overall, all approaches are better adapted to low-motion video content.

On the other hand, on average, 43% time saving is reduced by [LZZ + 19] with an increase in BD-BR of 2.56% and a decrease in BD-PSNR of -0.099dB. The proposed method presented in [TTAA19] allows 54.57% encoding time while the BD-BR increases by 2.97% and the BD-PSNR degradation reaches -0.107dB. Regarding the work presented in [XLW + 18], the proposed method surpasses ours in terms of BD-BR and BD-PSNR, while our proposed approach allows a significant coding time saving of 58.60% compared to this work. When comparing our work to the state of the art schemes ], and [XLW + 18] describing earlier, we can conclude that the proposed CNN-LSTM-based learning method proves the best coding efficiency of HEVC at inter-mode in order to predict the CU partition.

[LZZ + 19], [ TTAA19

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY

II.5 Conclusion

In this chapter, we proposed a fast CU partition based on machine learning approaches to reduce the HEVC complexity of inter-mode. An online SVM-based fast CU partition method was proposed to reduce the encoding complexity of HEVC. Then, to predict the CU partition of HEVC, a Deep CNN was proposed, which reduces the HEVC complexity at inter-mode. Unfortunately, these two machine learning algorithms do not explore the correlation of the CU partition across neighboring frames. Therefore, a deep learning approach was proposed to predict the CU partition at inter-mode, which combines the CNN and the LSTM structures. Simulations results show and prove the efficiency of the proposed framework in saving a significant encoding complexity, compared to other previous approaches-based machine learning tools.

In the next chapter, we provide a deep learning technique to improve the visual quality of reconstructed video for the VVC standard in order to meet the demands required by the users. we evaluate the proposed method in section III.5. Finally, section III.6 refers to the conclusion of this chapter.

III.2 Background

The growing multimedia portfolio, including Big Data processing, Cloud Computing and the IoT [MBB + 20], has a direct impact on our lifestyle. M-IoT is considered as a major network technology enabling the interconnection and interaction between humans, health-centers, industries, and objects like cameras, transport, and sensors [START_REF] Cao | A survey of emerging m2m systems: Context, task, and objective[END_REF].

In addition, M-IoT systems combine the networking technologies for computer vision, image processing, and connectivity. Yet, they can be used in driving assistance, surveil-

QoE BR = a × log(BR) + b (III.1)
where a and b denote coefficients determined during the experiment. However, this parameter, just like the PSNR metric, will also be used for the proposed WSE-DCNNbased in-loop filtering to evaluate video quality.

III.4 Proposed Method

This section introduces the proposed method and describes the WSE-DCNN architecture and how this is integrated into VVC standard to replace the traditional loop filtering technique in order to improve the video quality.

CHAPTER III. DEEP LEARNING BASED VIDEO QUALITY ENHANCEMENT FOR THE NEW VERSATILE VIDEO CODING

otherwise the flag is disabled. After all the CTUs in one frame are determined, the framelevel RD cost before and after filtering are calculated using equation III.2 indicated by J1 and J2, respectively. If J1 > J2, the frame-level flag will be enabled. Hence the corresponding frame-level flag can be encoded in the slice header and CTU-level control flags can be signaled into each corresponding CTU syntax. Otherwise, the frame-level flag is disabled and CTU-level flags will not be encoded for transmission anymore.

III.4.2 WSE-DCNN Architecture

The concept of the proposed architecture is illustrated in The WSE unit consists of the following phases as depicted in Algorithm 1, given a feature map X with shape H × W × C, where C means channel amounts:

• A wide 3 × 3 convolution followed by ReLU and a convolution layer with kernel size is 1 × 1. Given Y 1 is the channel defined in Algorithm 1 and Y 2 is the output of the second convolution layer.

• Each channel obtains a value according to the squeeze operation using Global Average Pooling (GAP) Y 3 (k) as shown in Algorithm 1.

• The excitation operation is described by two fully connected layers followed by ReLU and sigmoid (σ) activation functions, respectively. As shown in Algorithm 1, Y 4 is the first fully connected layer followed by ReLU, which is refined by a certain • According to WSE function, each Y 2 channel is multiplied by the gating ratio r, as defined Algorithm 1.

Y 1 = ReLU (W 1 X + b 1 ) Y 2 = W 2 Y 1 + b 2 return (Y 2 ) Squeeze-
• Finally, when the number of input equals to the output channels C, a skip connection will be added directly from input to output to learn the residue. Otherwise, there is no skipped connection.

L(θ) = 1 N N i=1 ||F (Y i , θ) -X i || 2 2 (III.5) Let X i is

III.5.4 Comparative Study

We also compared the proposed approach with other filtering models based on CNN network. Deep learning algorithms, such as CNNs, have significant higher accuracies than traditional algorithms, but they require huge amounts of computational resources and memory access due to the large number of parameters in the layers operation, which represents a computational challenge. Therefore, many hardware accelerators, such as Field Programmable Gate Arrays (FPGAs), especially the new technology FPGA-SoC, are considered as the most promising platforms for accelerating CNNs, due to their high performance capabilities, energy efficiency, and reconfigurable property.

This chapter provides a hardware-software architecture based on an accelerated CNN model for a video compression application. We first accelerate the CNN layers to build an Intellectual Property (IP) cores using Vivado High Level Synthesis (HLS). Then, we create a hardware-software architecture based on a CNN's IP cores designed and integrated in the Programmable Logic zone (PL) which is connected to the Xilinx Processing System (PS) that manage all processing tasks on the FPGA-SoC board.

The remainder of this chapter is organized as follows. Section IV.2 provides the background in which the preliminary study are included. Section IV.3 discusses the proposed CNN accelerator on FPGA-SoC. Section IV.4 describes the experimental results and the discussions. Section IV.5 concludes this chapter.

IV.2 Background

Recent AI methods, such as deep learning, have enjoyed considerable success in various machine learning tasks because of their powerful learning ability [START_REF] Kang | Energy efficiency of machine learning in embedded systems using neuromorphic hardware[END_REF]. They have been broadly applied in many signal processing areas including computer vision, image processing, data mining. However, computationally intensive deep learning algorithms had to be run on embedded devices [START_REF] Hong | Design of power-efficient training accelerator for convolution neural networks[END_REF], such as FPGAs. Especially, the new technology FPGA-SoC are considered as the most promising platforms for accelerating AI methods, due to their real-time performance, high energy efficiency and flexible designs.

CHAPTER IV. DEEP CNN CO-DESIGN FOR HEVC CU PARTITION PREDICTION ON FPGA-SOC

(AXI) connections.

IV.2.1.1 Processing System (PS)

All Zynq devices have the same basic architecture, and all of them contain, as the basis of the processing system, a dual-core ARM Cortex-A9 processor [START_REF] Crockett | The Zynq Book: Embedded Processing with the Arm Cortex-A9 on the Xilinx Zynq-7000 All Programmable Soc[END_REF]. eral, the advantage of soft processors is that the number and precise implementation of processor instances are flexible. On the other hand, hard processors can achieve higher performance, as is the case with Zynq's ARM processor.

It is important to note that the Zynq processing system encompasses not only the ARM processor, but a set of associated processing resources forming an application processing unit, as well as other peripheral interfaces, cache memory, memory interfaces,

IV.2.1.2 Programmable Logic (PL)

PL is the second principal part of the Zynq architecture [START_REF] Crockett | The Zynq Book: Embedded Processing with the Arm Cortex-A9 on the Xilinx Zynq-7000 All Programmable Soc[END_REF], which is based on the Artix®-7 and Kintex®-7 FPGA fabric. The PL part of the Zynq device is illustrated in

IV.2.2 Direct Memory Access (AXI DMA)

AXI DMA transfers data between memory and AXI4-Stream-type target peripherals

IV.2.3 FPGA-SoC: PYNQ-Z1

The PYNQ-Z1 FPGA is the chosen hardware platform, which is based on Xilinx ZYNQ SoC technology [START_REF]PYNQ Xilinx. Python productivity for zynq[END_REF]. It provides a Python environment, to make it easier for designers to exploit the PL and PS of the FPGA board. Xilinx offers Python packages and associated libraries to facilitate the interaction with hardware modules based on Overlays. Overlays, or hardware libraries, are designed to be programmable and reusable FPGA designs to extend the user application from The PS into the PL of the ZYNQ.

An overlay is a PL design class developed by hardware designers. PYNQ overlays can be customized the hardware platform for a certain application.

IV.4.2 Hardware Cost of the Proposed Co-Design

This section presents the percentage of hardware resources consumed by the proposed co-design on the PYNQ-Z1 platform. After the implementation phase, the hardware resource occupancy of our proposed design is shown in Table IV.4. FFs, and more than 25% BRAMs. This design achieves an on-chip power consumption of 15.8 W under a 120 MHz working frequency of the FPGA. In reference [START_REF] Zhang | Fpga implementation for cnn-based optical remote sensing object detection[END_REF], an efficient hardware-implementation method for optical remote sensing object detection was proposed. However, this design has extremely high requirements for resource utilization, consuming more than 70% BRAMs. It achieves 5.96 W on-chip power consumption at a clock frequency of 200 MHz. In addition, the authors in [LZF + 19] proposed a CNN accelerator, which accelerates the standard convolution and the depthwise separable convolution. This method has been implemented on the Xilinx ZYNQ 7100 hardware platform which achieves an on-chip power consumption of 3.99W at a clock frequency of 100MHz. Meanwhile, it consumed the highest hardware resources, using almost all DSPs, 50% LUTs, and over 40% BRAMs.

After this comparative study, we remark that our proposed design achieves high performance; low power consumption around 1.69W and occupies low hardware FPGAs resources. Therefore, our design can achieve a satisfactory balance between resource cost and power consumption and it is suitable for deployment on embedded devices with limited resource budget. The future work will be explored to accelerate the overall architecture of Deep CNN and implement it in a real-time on PYNQ-Z1 using customized Overlay.

General Conclusion and Perspectives

The expansion of Internet coupled with the rapid introduction of UHD, HDR and 360°v ideo contents in daily life have caused the explosion of video traffic. Recent study published in Cisco [START_REF] Cicero | Cisco predicts more ip traffic in the next five years than in the history of the internet[END_REF] has predicted that video traffic will increase from 75% of the global IP traffic in 2017 to 82% in 2022. This increasing demand for video contents brings new challenges to compression, especially to enhance the coding efficiency and enable a high QoE of video services. Driven from these requirements, various technologies appeared as potential solutions for video coding deployment. In this thesis work, the main concern is reducing computational complexity and improving video quality for HEVC and VVC standards using artificial intelligence.

In this thesis, we have focused on video coding standards, such as HEVC and VVC.

Particularly, we tackled the problem of complexity reduction and video quality using machine learning solutions. Four main contributions have been integrated in this work.

Firstly, we have reviewed the basic building blocks of a video compression system. where in the last line we have used the fact that ∂b l j ∂z l j = 1. This is the second of the four backpropagation equations. We now derive the final two backpropagation equations using the chain rule. Since the error depends on neurons in layer l only through the

I. 1

 1 History of Video Coding Standardization . . . . . . . . . . . . . . . . . . I.2 Block Diagram of the Hybrid Video Coding Layer for HEVC . . . . . . . I.3 HEVC Quadtree Partitioning Structure, including CU, PU and TU (solid line for CU, dashed line for TU) . . . . . . . . . . . . . . . . . . . . . . . I.4 Intra Prediction Modes in the HEVC Standard . . . . . . . . . . . . . . . I.5 Example of Uni and Bi-directional Inter Prediction . . . . . . . . . . . . . I.6 Example of Quadtree with Nested Multi-Type Tree Coding Block Structure for VVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I.7 Intra Directional Modes in VVC . . . . . . . . . . . . . . . . . . . . . . . I.8 Wide-Angular Intra-Picture Prediction . . . . . . . . . . . . . . . . . . . . I.9 Example of HEVC Time Profile . . . . . . . . . . . . . . . . . . . . . . . . I.10 Example of SVM Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . I.11 Overall Convolutional Neural Network Architecture . . . . . . . . . . . . II.1 CU Partition Structure in HEVC . . . . . . . . . . . . . . . . . . . . . . . II.2 Flowchart of the Proposed Algorithm . . . . . . . . . . . . . . . . . . . . . II.3 Online Training Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . II.4 Deep CNN Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . II.5 Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . II.6 Proposed Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . II.7 LSTM Cell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . II.8 Learning Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . II.9 Encoding Time of the Proposed CNN-LSTM and Deep CNN . . . . . . . III.1 M-IoT Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III.2 M-IoT Scenario-based Centralized Video Quality Enhancement . . . . . . III.3 Proposed VVC Standard Framework . . . . . . . . . . . . . . . . . . . . . III.4 WSE-DCNN Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . III.5 Sample Frames of Sequences from the BVI-DVC Database . . . . . . . . . III.6 Training MSE Loss and Validation PSNR . . . . . . . . . . . . . . . . . . xii LIST OF FIGURES III.7 Ablation Study. Subjective Visual Quality Comparison (the 12th frame of BQSquare with QP =37: (a) Original; (b) VVC without in-loop filtering (P SN R=31.17dB); (c) VVC (P SN R=31.37dB); (d) VVC-based proposed model (P SN R=31.68dB) . . . . . . . . . . . . . . . . . . . . . . III.8 Comparison of QoE Variation with Respect to bitrate . . . . . . . . . . . III.9 RD-performance Curves of the Proposed Model Compared to other three Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV.1 A simplified Model of the Zynq Architecture . . . . . . . . . . . . . . . . . IV.2 Zynq Processing System . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV.3 Zynq Programmable Logic . . . . . . . . . . . . . . . . . . . . . . . . . . IV.4 Block CLB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV.5 AXI Interconnects and Interfaces . . . . . . . . . . . . . . . . . . . . . . . IV.6 Vivado HLS Design Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . IV.7 Proposed Deep CNN-based CU Partition for HEVC . . . . . . . . . . . . IV.8 CONV-IP Cores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV.9 FC-IP Cores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV.10Hardware-Software Co-Design . . . . . . . . . . . . . . . . . . . . . . . . . IV.11Power Consumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1 Convolution Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 The Non-Linear Activation Functions . . . . . . . . . . . . . . . . . . . . A.3 Max vs Average pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.4 Fully Connected Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii List of Tables I.1 Coding Tools of VVC vs HEVC . . . . . . . . . . . . . . . . . . . . . . . . II.1 Sequences in CPIH Database . . . . . . . . . . . . . . . . . . . . . . . . . II.2 Test Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . II.3 Performances Comparison between Deep CNN and Online SVM . . . . . II.4 Performances Comparison between Deep CNN and CNN-LSTM . . . . . . II.5 Comparative Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III.1 Key Features of BVI-DVC Video Training Database [MZB20] . . . . . . . III.2 Performance Evaluation of the Proposed model under RA Configuration . III.3 Coding Performance Comparison with other Approaches . . . . . . . . . . IV.1 CNN_1 Model Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . IV.2 Hardware Resource Occupation of the CONV-IP . . . . . . . . . . . . . . IV.3 Hardware Resource Occupation of the FC-IP . . . . . . . . . . . . . . . . IV.4 Hardware Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV.5 Comparative Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv General Introduction

  Learning Approach-based Fast CU Partition for Reducing HEVC Complexity The second chapter proposes a fast Coding Unit (CU) partition based on machine learning approaches to reduce the HEVC complexity of inter-mode. An online Support Vector Machines (SVM)-based fast CU partition method is proposed for reducing the encoding complexity of HEVC. Afterwards, to predict the CU partition of HEVC, a Deep Convolutional Neural Network (CNN) is proposed, which reduces the HEVC complexity at inter-mode. Unfortunately, these two machine learning algorithms do not explore the correlation of the CU partition across neighboring frames. A Long-and Short-Term Memory (LSTM) model was developed to learn the temporal dependency of the intermode CU partition. Therefore, a deep learning approach is proposed to predict the CU partition at inter-mode, which combines the CNN and LSTM structures. Finally, the obtained results are discussed in order to evaluate the performance of the proposed algorithms. Chapter III : Deep Learning based Video Quality Enhancement for the New Versatile Video Coding General Introduction The third chapter proposes a deep learning algorithm-based VVC standard to enhance visual video quality while improving the user's Quality of Experience (QoE). The proposed Wide-activated Squeeze-and-Excitation Deep Convolutional Neural Network (WSE-DCNN) model is integrated into VVC standard to replace in-loop filtering in order to alleviate the coding artifacts, such as ringing, blocking, and blurring. The proposed VVC filtering technique is used in the Multimedia-Internet of Things (M-IoT) scenario-based smart city context to help the centralized cloud meet the user's required video quality. Finally, all simulation results obtained are interpreted and compared to the related existing methods. Chapter IV : Deep CNN Co-Design for HEVC CU Partition Prediction on FPGA-SoC The last chapter proposes a deep CNN based hardware-software design for HEVC CU prediction on FPGA-SoC. Our proposed work aims to accelerate the CNNs due to their computationally intensive. Hence, we create a hardware Intellectual Property (IP) core for each CNNs layer using the Vivado HLS tool. Then, we have designed a hardwaresoftware architecture by importing the hardware IP cores based on the PYNQ-Z1 board.

Summary I. 1

 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I.2 Video Compression History . . . . . . . . . . . . . . . . . . . . I.3 HEVC Standard . . . . . . . . . . . . . . . . . . . . . . . . . . . I.3.1 Sampled Representation of Pictures . . . . . . . . . . . . . . . I.3.2 Block Partitioning in the HEVC Standard . . . . . . . . . . . . I.3.3 Intra Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . I.3.4 Inter Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . I.3.5 Transform and Quantization . . . . . . . . . . . . . . . . . . . I.3.6 Entropy Coding . . . . . . . . . . . . . . . . . . . . . . . . . . I.3.7 In-Loop Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . I.4 VVC Standard . . . . . . . . . . . . . . . . . . . . . . . . . . . . I.4.1 Block Partitioning in the VVC Standard . . . . . . . . . . . . . I.4.2 Intra Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . I.4.3 Inter Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . I.4.4 Transform and Quantization . . . . . . . . . . . . . . . . . . . I.4.5 Entropy Coding . . . . . . . . . . . . . . . . . . . . . . . . . . CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS I.4.6 In-Loop Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 I.5 Video Coding Challenges . . . . . . . . . . . . . . . . . . . . . . 15 I.6 Artificial Intelligence: New Advancements and Innovations . 16 I.6.1 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . 16 I.6.2 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 I.7 Related Research . . . . . . . . . . . . . . . . . . . . . . . . . . 23 I.7.1 Heuristic Methods . . . . . . . . . . . . . . . . . . . . . . . . . 23 I.7.2 Learning Methods . . . . . . . . . . . . . . . . . . . . . . . . . 24 I.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

  Historically, two major video coding standardization organizations have coexisted: Moving Picture Experts Group (MPEG), which belongs to the International Organization for Standardization and the International Electrotechnical Commission (ISO/IEC), and the Visual Coding Experts Group (VCEG), which belongs to the International Telecommunication Union, Telecommunication Standardization Sector (ITUT). The history of the different H.26x and MPEG-x families established by ITUT and ISO/IEC is shown in Figure I.1.

Figure

  Figure. I.1: History of Video Coding Standardization

  over H.264/AVC in order to achieve better compression efficiency. The block diagram of a hybrid video coding layer conforming with the HEVC standard is illustrated in Figure I.2.

Figure

  Figure. I.2: Block Diagram of the Hybrid Video Coding Layer for HEVC

Figure. I. 4 :

 4 Figure. I.4: Intra Prediction Modes in the HEVC Standard

Figure. I. 5 :

 5 Figure. I.5: Example of Uni and Bi-directional Inter Prediction

Figure I. 3 ,

 3 TUs of size 4×4, 8×8, 16×16 and 32×32 are supported. The 2D transforms based on Discrete Cosine Transform (DCT) are designed for them and special efforts are particularly spent on selecting the value of the transfrom matrix for retaining the property of easy-to-implementation [SOHW12]. In addition, when transforming for 4×4 block size in intra-frame prediction mode, another integer transformation based on Discrete Sine Transform (DST) is available for use. The resulting transform coefficients are then quantized, before being sent to the construction of the coded bitstream. Quantization is a compression technique which converts a range of values into a single quantum value. The maximum Quantization Parameter of HEVC standard is set to 51.

Figure I. 6

 6 Figure I.6 shows one CTU divided into multiple CUs with a QuadTree plus Multi-

65.

  The new directional modes, not used in HEVC, are depicted by red dotted arrows, as mentioned in Figure I.7, whereas the planar and DC modes are unchanged for both video encoder. These denser directional intra prediction modes apply for all block sizes andCHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDSfor both luma and chroma intra predictions. In HEVC, 33 angular prediction directions are defined from 45°to 135°in a clockwise direction. In VVC, the angular precision is basically doubled to produce 65 angles within that same range, and another 28 "wideangle" prediction modes beyond this angular range can be used for non-square blocks, as illustrated in Figure I.8.

Figure. I. 7 :

 7 Figure. I.7: Intra Directional Modes in VVC

  coding blocks. The newly introduced transformation matrices are DST-VII and DCT-VIII. The change of the Quantization stage is the increase in the maximum Quantization Parameter (QP) from 51 to 63.

  Figure. I.9: Example of HEVC Time Profile

  The hyperplane, where possible, classifies or separates the data correctly while being as far as possible from all observations, based on the training set. The principle is therefore to find a classifier or a discrimination function whose generalization capacity (forecast quality) is acceptable for the specific application. Therefore, the purpose of SVM is the reduction of discrimination problem to the linear problem of finding an optimal hyperplane [HCP+ 18]. In Figure I.10, the principle of the SVM algorithm has been shown. Finding the optimal hyperplane to differentiate classes is the major functionality of SVM techniques. The Figure I.10 (a) presents two classes consisting of circles and stars which need to be separated. SVM is a frontier which best segregates the two classes (hyperplane). Now the important question is how can one identify the right hyperplane?. The response is in the Figure I.10 (b) which maximizes the distance between the nearest data point (either class) and hyperplane will help us to decide the right hyperplane. This distance is defined as margin. So, the margin of the hyperplane C is the highest as compared to A and B. Hence, the hyperplane C is the optimal hyperplane that can classify data set.

Figure

  Figure. I.10: Example of SVM Classifier

  [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] applied a gradient-based learning algorithm to CNNs and obtained successful results for the handwritten digit classification problem. After that, researchers further improved CNNs and reported state-of-the-art results in many recognition tasks. CNNs have several advantages over Deep Neural Networks (DNNs), including being more like the human visual processing system, being highly optimized in the structure for processing 2D and 3D images, and being effective at learning and extracting abstractions of 2D features.

Figure I. 11

 11 Figure I.11 introduces the overall architecture of CNNs consisting of two main parts:Feature extraction and classification. In the feature extraction layers, each layer of the network receives the output from its immediate previous layer as its input and passes its output as the input to the next layer. In the classification part, the feature maps of

  CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDSwell on training data that it negatively impacts the performance of the model on new data.In order to overcome the problem of overfitting, a dropout layer can be introduced in the model in which some neurons along with their connections are randomly dropped from the network during training. Only the reduced network is trained on the data in that stage. The removed nodes are then reinserted into the network with their original weights. Dropout notably reduces overfitting and improves the generalization of the model.

  4. It squashes the input into the range [0, 1]. σ (x) = 1 1 + e -x (I.4) In addition, the hyperbolic tangent function (tanh) is similar to sigmoid function but its output lies in the range [-1, 1]. The advantage of tanh over sigmoid is that the negative inputs will be mapped strongly negative and the zero inputs will be mapped near zero. Moreover, softmax function is often used in the output layer of a neural network for classification. It is a more generalized logistic activation function which is CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS used for multiclass classification, which is defined as. σ (x j ) = e x j n k=1 e x k (I.5) I.6.2.5 Backpropagation Algorithm For the training phase, the common algorithm used is the backpropagation [RZ85]. The training procedure requires us to be able to calculate the derivative of the cost function with respect to all the parameters of the neural network (the weights and biases of all the neurons in the input, hidden, and visible layers). The backpropagation algorithm is a clever procedure that exploits the layered structure of neural networks to more efficiently compute gradients [MBW + 19]. This algorithm consists of a forward pass from the bottom layer to the top layer where one calculates the weighted inputs and activations of all the neurons. One then backpropagates the error starting with the top layer down to

  [CMMC19, WLM + 16, XLWM13, CK13, FSK + 20, PK19]. To reduce the HEVC computational complexity, authors in [CMMC19] introduced a look-ahead stage-based fast partitioning and CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS mode decision algorithm. Wang et al. in [WLM + 16] proposed a threshold-based splitting decision scheme with respect to the RD cost of each CU. It reduces the number of available intra candidates, adaptive reference frame selection and early termination of coding unit splitting. In [XLWM13], authors proposed a fast algorithm to split CU based on pyramid motion divergence at inter prediction. In addition, a fast early CU-splitting and pruning method with low complexity and full RD cost was developed by Cho et al.in[START_REF] Cho | Fast cu splitting and pruning for suboptimal cu partitioning in hevc intra coding[END_REF]. In a similar way, authors in [FSK+ 20] proposed a fast QTMT partition algorithm based on variance and gradient to reduce the computational complexity brought in by the novel MT partitions in VVC. Reference[START_REF] Park | Context-based ternary tree decision method in versatile video coding for fast intra coding[END_REF] proposes a context-based ternary trees (TT) decision (C-TTD) method to significantly reduce TT computational complexity in VVC intra-coding. These methods are based on the statistics on the RD cost

  scheme for VVC intra encoders. Similarly, reference [LXT + 21] proposes a deep learning approach to predict the QTMT-based CU partition, for drastically accelerating the encoding process of intra-mode VVC. A fast CU partition decision algorithm based on the improved Directed Acyclic Graph Support Vector Machine model to reduce the complexity of CU partition [ZWH + 21]. In addition, other components of HEVC and VVC, such as in-loop filtering, are simplified to reduce the encoding complexity.

  Figure. II.1: CU Partition Structure in HEVC

  Figure. II.2: Flowchart of the Proposed Algorithm

Figure. II. 3 :

 3 Figure. II.3: Online Training Mode

Figure. II. 4 :

 4 Figure. II.4: Deep CNN Architecture

Figure II. 5 .

 5 Figure II.5. We train our model in a supervised learning manner, in which the Deep CNN has been learned based on labeled data. In this context, we create the database for training the proposed model, which satisfy highly performances (high accuracy and low loss). Afterwards, we establish a large-scale database for CU partition of the inter-mode HEVC (CPIH), in order to increase the prediction accuracy. However, to construct our CPIH database, we selected 114 raw video sequences with various resolutions from 352 × 240 to 2560×1600 [XDLW14, OSS + 12, ML94, B + 13]. These sequences are gathered into three sub-sets: 86 sequences for training, 10 sequences for validation, and 18 sequences for test. Table II.1 summarizes the chosen videos and the number of frames (41, 349) in our CPIH database. First, we encoded the original database (114 video sequences) by original HEVC encoder common test condition at different Quantization Parameters (QP=22, 27, 32, 37) using Low Delay P configuration (using encoder -lowdelay -P -main.cf g ) to obtain the residue and the ground truth CU depth. The ground truth CU depth files

  Figure. II.6: Proposed Framework

  , as shown in Figure II.6. The ReLU and the sigmoid activation functions are used to activate the hidden and the output layers, respectively [GBB11]. When predicting CTU partition, the long short-term dependency of CTU partition across frames can be taken into consideration in the LSTM network. As seen in Figure II.6, the temporal dependency is modeled by the LSTM cells, which are processed along with the encoded frames. Here, we take the LSTM cell of level l at frame t as an example to discuss the internal mechanism of the proposed LSTM. In fact, the LSTM cell consists of three gates, as shown in Figure II.7; the input gate i l (t), the forget gate f l (t), and the output gate o l (t).

  Figure. II.7: LSTM Cell

  Figure. II.8: Learning Process

  RD performance analysis is performed based on the Bjontegaard Delta bitrate (BD-BR) and the Bjontegaard Delta Peak Signal-to Noise Ratio (BD-PSNR) [Bjo01]. The BD-BR represents the average bitrate savings that calculated between two RD curves for the same video quality, where negative BD-BR values indicate actual bitrate savings and positive values indicate how much the bitrate is increased. BD-PSNR is the overall PSNR difference of RD curves with the same bitrate in decibel. Not forgetting that the coding time is modeled as the critical metric for the validation performance of the HEVC at inter-mode, as shown in the following equation:

Figure

  Figure. II.9: Encoding Time of the Proposed CNN-LSTM and Deep CNN

  (M-IoT) is an emerging type of Internet of Things (IoT) relaying multimedia data (image, video, audio and speech, etc...) [NQA + 20]. The rapid growth of M-IoT devices enables the creation of a massive volume of multimedia data with different characteristics and requirements [MBB + 20]. With the development of Artificial Intelligence (AI), AI-based M-IoT systems have been recently designed and deployed for various video-based services for contemporary daily life, like video surveillance with HD and UHD and mobile multimedia streaming. These new services need higher video quality in order to meet the Quality of Experience (QoE) required by users [ARM18]. This chapter proposes a deep CNN-based in-loop filtering approach, denoted as the Wide-activated Squeeze-and-Excitation Deep Convolutional Neural Network (WSE-DCNN). The proposed approach provides new powerful in-loop filtering without exploiting traditional ones for the VVC standard. Indeed, the main goal is to effectively remove compression artifacts and enhance the compressed video quality. The proposed method improves the QoE of end-users. The remainder of this chapter is organized as follows. Section III.2 presents the background. Section III.3 introduces the proposed M-IoT scenario. Then, the proposed deep CNN-based in-loop filtering in VVC standard is defined in section III.4. Next,

CHAPTERFigure

  Figure III.1. However, several issues, such as interoperability, security, data size, reliability, storage and computational capacity need to be well resolved to process multimedia data [ZKH + 19].

Figure .

 . Figure. III.4: WSE-DCNN Architecture

CHAPTER

  III. DEEP LEARNING BASED VIDEO QUALITY ENHANCEMENT FOR THE NEW VERSATILE VIDEO CODING Algorithm 1 WSE-Unit Input: X ∈ {H, W, C} Output: Y ∈ {H, W, C} 1 for number of Epochs do 2

  2 (i, j, k). return(Y 3 ) Call-Excitation-Operation(Y 3 ): Y 4 = ReLU (W 4 Y 3 + b 4 ). Y 5 = σ(W 5 Y 4 + b 5 ). return(Y 5 ) 2 , Y 5 ): Y 6 (i, j, k) = Y 2 (i, j, k) × Y 5 (k), ∀i ∈{1, ..., H}, ∀j ∈ {1, ..., W }, ∀k ∈ {1, ..., C}. return(Y 6 ) ratio r. Then, the second fully connected layer followed by the sigmoid activation function which is denoted by Y 5 , and it gives each channel a smoothing gating ratio in the range of [0,1].

Features

  Figure. III.5: Sample Frames of Sequences from the BVI-DVC Database

  Figure. III.7: Ablation Study. Subjective Visual Quality Comparison (the 12th frame of BQSquare with QP =37: (a) Original; (b) VVC without in-loop filtering (P SN R=31.17dB); (c) VVC (P SN R=31.37dB); (d) VVC-based proposed model (P SN R=31.68dB)

  Figure. III.8: Comparison of QoE Variation with Respect to bitrate

Figure

  Figure. III.9: RD-performance Curves of the Proposed Model Compared to other three Approaches

FPGAs have been used

  to improve CNN performance, which is the purpose of the next chapter. are widely used, due to their excellent performance, in many computer vision applications, such as facial recognition, image classification tasks, speech recognition programs, video gaming, etc. However, CNNs require a large number of memory resources and they are also computationally intensive.

  Figure. IV.2: Zynq Processing System

CHAPTER

  IV. DEEP CNN CO-DESIGN FOR HEVC CU PARTITION PREDICTION ON FPGA-SOC interconnect, and clock generation circuitry [CEES14].

Figure

  Figure IV.3, with various features highlighted. The PL is mainly composed of a general purpose FPGA logic structure, which is made up of slices and Configurable Logic Blocks (CLBs), as well as Input/Output Blocks (IOBs) for interfacing. Indeed, CLBs are small, regular groupings of logic elements that are laid out in a two-dimensional array on the PL, and connected to other similar resources via programmable interconnects.

Figure. IV. 3 :

 3 Figure. IV.3: Zynq Programmable Logic

[

  Joh14]. AXI DMA in Vivado provides high-bandwidth direct memory access between an AXI4 memory-mapped and an AXI4-Stream ports on IPs interfaces [Xil19a]. PYNQ supports the AXI central DMA IP with the PYNQ DMA class [PYN19]. DMA can be used for high performance burst transfers between PS DRAM and PL. It helps to offload data from the Central Processing Unit (CPU) in processor-based systems [Xil19a]. AXI DMA data movement between system memory and stream target is through the AXI4 Read Master to AXI4 memory-mapped to stream (MM2S) Master, and AXI stream to memory-mapped (S2MM) Slave to AXI4 Write Master.

CHAPTER

  IV. DEEP CNN CO-DESIGN FOR HEVC CU PARTITION PREDICTION ON FPGA-SOCLUT slices consumed by our three accelerators, including FC1-IP, FC2-IP and FC3-IP is 3%. The hardware cores use 1% of FFs and 2% of DSPs. In addition, 1% of BRAMs are mainly used by the FC1-IP accelerator. These IPs operate at a frequency of 150 MHz.

  In this chapter, we have proposed a deep CNN based hardware-software design for HEVC CU prediction on FPGA-SoC. Our proposed work aims to accelerate the CNNs due to their computationally intensive. However, we have created a hardware IP core for each CNNs layer using the Vivado HLS tool. Then, we have designed a hardware-software architecture by importing the hardware IP cores based on the PYNQ-Z1 board. Compared to other designs, our architecture has shown clear advantages in terms of power consumption and hardware resources, which is suitable for deployment on embedded devices with limited resources.

A

  scheme saved a significant encoding complexity, compared to other previous approachesbased machine learning tools.

CHAPTER I. VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS used

  in the HEVC standard, in which the analogous structure, called Coding Tree Unit

			PU	2Nx2N	2NxN	Nx2N	NxN	2NxnU	2NxnD	nLx2N	nRx2N
	CTU
				TU					TU
					CTU
				4		5
	0 1	2	3					6	7	8
										9	10 11 12
	Figure. I.3: HEVC Quadtree Partitioning Structure, including CU, PU and TU (solid
	line for CU, dashed line for TU)
	coding efficiency, where available block size varies from 4×4 up to 64×64, including
	symmetric partitioning, such as 2N×2N, 2N×N, N×2N and N×N, and also asymmetric
	motion partitioning (AMP) for instance 2N×nU, 2N×nD, nL×2N and nR×2N. In par-
	ticular N×N is only allowed for minimum coding unit (CU) and AMP is not applied to
	CUs smaller than 16×16. Figure I.3 introduces the partitioning and quadtree structure

Table I

 I

.1 summarizes the main coding tools of HEVC and VVC. VVC has adopted many new coding tools in each coding stage [MMS + 21].

VIDEO CODING AND ARTIFICIAL INTELLIGENCE BACKGROUNDS to

  handle a large amount of data but difficult to build a good model which is able to effectively recognize new objects in a new test. Machine learning (ML) is an attempt to understand and reproduce this learning facility in an artificial system. It therefore seems appropriate to use techniques from this field to discover and model knowledge and reduce the semantic gap[START_REF] Mitchell | An artificial intelligence approach[END_REF]. ML is at the crossroads of various fields such as artificial intelligence, statistics, cognitive science, probability theory, optimization, signal and information, and so on [B + 01] [RN16][START_REF] Fulcher | Computational intelligence: an introduction[END_REF]. It is therefore very difficult to give taxonomy of machine learning categories. Then, we briefly present in this section the four main types of machine learning techniques [DB17]: Supervised Learning [KZP07],

One generally distinguishes the learning which consists of memorizing information [AM05] [SS94], and the learning by generalization [Wit74] [OW83] in which we usually build a model from learning examples to recognize new examples and scenarios. For the machines, it is easy CHAPTER I. Unsupervised Learning [HTF09], Semi-supervised Learning [CSZ09], and Reinforcement Learning [KLM96].

Table II

 II

			.1: Sequences in CPIH Database
	Resolutions		Train Data		Valid Data		Test Data
		N.	N.	N.	N.	N.	N.
		of video of frame of video of frame of video of frame
	352x240(SIF)	4	677	-	-	-	-
	352x288(CIF)	23	6,530	2	550	-	-
	704x576(4CIF)	4	2,280	1	600	-	-
	720x486(NTSC)	6	1,800	1	300	-	-
	416X240(240p)	-	-	-	-	4	1,900
	832x480(480p)	-	-	-	-	4	1,900
	1280x720(720p)	5	1,327	2	1,100	3	1,800
	1920x1080(1080p)	28	8,417	2	540	5	2,080
	2048x1080(2k)	16	8,048	2	1,200	-	-
	2560x1600(WQXGA) -	-	-	-	2	300
	Total	86	29,079	10	4,290	18	7,980

where N is the number of training samples and Ln represents the sum of the cross entropy:

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY each

  CU. Specifically, F1 (CU, t) at level 1 indicating whether the 64 × 64 size CU will be split into 32 × 32 size sub-CUs or not. At level 2, { F2 (CU i , t)} 3 i=0 and { F3 (CU i,j , t)} 3At each level, two fully connected layers that contains a hidden layer and an output layer are followed the LSTM cells. In addition, the output features of the LSTM cells are denoted by (F C l ) 3 l=1 at frame t. If the CU of the current level is predicted to be split, the LSTM classifier of the next level is activated to make decisions on the four subsequentCUs at the next level. Otherwise, the prediction on partitioning the current CTU is terminated, in order to save computational time. Finally, the CU splitting results of three levels are combined to represent the CTU partition in the form of 21-dimensional

	i,j=0
	designate respectively the CU partition labels from 32 × 32 to 16 × 16 and from 16 × 16
	to 8 × 8.

vector, which is composed of F1 (CU, t), { F2 (CU i , t)} 3 i=0 and { F3 (CU i,j , t)} 3 i,j=0

Table II

 II

			.2: Test Sequences
	Class	Resolutions	Sequences
	A	2560 × 1600	PeopleOnStreet, Traffic
	B	1920 × 1080	Kimono, ParkScene, Cactus, BQTerrace, BasketballDrive
	C	832 × 480	BasketballDrill, BQMall, PartyScene, RaceHorses
	D	416 × 240	BasketballPass, BQSquare, BlowingBubbles, RaceHorses
	E	1280 × 720	FourPeople, Johnny, KristenAndSara

MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY performance

  TableII.3 gives a comparison of our two proposed methods, Deep CNN and Online SVM, in terms of complexity reduction and RD performance using LDP configuration. of inter-mode HEVC, as seen in TableII.3. This implies that the proposed Deep CNN is robust in reducing complexity of inter-mode HEVC when compared to the online SVM. This is due to the fact that the CNN works well with visual images recognition while SVM is used widely in classification problems. Additionally, it is difficult to parallelize SVM but the CNN architecture inherently supports parallelization.

	CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU
	PARTITION FOR REDUCING HEVC COMPLEXITY
	Table II.3: Performances Comparison between Deep CNN and Online SVM
	Sequences		Online SVM		Deep CNN
		BD-BR BD-PSNR	T	BD-BR BD-PSNR	T
		(%)	(dB)	(%)	(%)	(dB)	(%)
	PeopleOnStreet 3.65	-0.154	-56.56	1.20	-0.051	-50.67
	Traffic	3.94	-0.106	-58.07	1.49	-0.041	-57.90
	Kimono	1.12	-0.036	-44.18	1.38	-0.044	-43.26
	ParkScene	1.67	-0.048	-52.60	1.43	-0.041	-64.14
	Cactus	1.68	-0.034	-41.38	2.44	-0.047	-52.57
	BQTerrace	1.67	-0.029	-41.45	2.22	-0.034	-58.43
	BasketballDrive 1.72	-0.039	-51.17	2.28	-0.051	-51.30
	BasketballDrill	2.72	-0.098	-55.87	1.43	-0.052	-53.54
	BQMall	5.11	-0.192	-55.96	2.24	-0.085	-52.25
	PartyScene	4.38	-0.169	-52.93	1.48	-0.057	-51.54
	RaceHorses	4.67	-0.173	-51.25	1.41	-0.053	-42.22
	BasketballPass	4.94	-0.217	-55.49	1.85	-0.083	-52.42
	BQSquare	6.63	-0.227	-55.92	2.09	-0.073	-52.79
	BlowingBubbles 4.56	-0.157	-51.87	1.71	-0.061	-46.55
	RaceHorses	7.48	-0.315	-50.81	1.32	-0.058	-38.01
	FourPeople	1.78	-0.054	-51.90	1.06	-0.029	-67.54
	Johnny	3.60	-0.076	-60.12	3.99	-0.083	-69.66
	KristenAndSara 2.51	-0.070	-53.57	1.31	-0.082	-67.20
	Overall	3.55	-0.121	-52.28 1.80	-0.057	-53.99
	As it can be seen, our proposed Deep CNN obtains significantly best results in terms of
	execution time for class E sequences, this is caused by the low motion activities displayed
	in these sequences, which leads to larger partitions. For the same reason, it is possible to
	The experimental results show that our Deep CNN model achieves a significant observe a slightly higher encoding time for high-resolution sequences compared to lower
	complexity reduction of around 53.99% with 1.80% BD-BR compared to the online SVM resolution ones.

at LDP configuration. On the other hand, our online SVM demonstrates significant coding losses in BD-BR of 3.55% and an average decrease of 52.28% in time reduction.

From the overall performance evaluation, we can find that the proposed method Deep CNN outperforms the online SVM in terms of both complexity reduction and RD CHAPTER II.

Table II

 II

	CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU
	PARTITION FOR REDUCING HEVC COMPLEXITY
		Proposed CNN-LSTM	T BD-BR BD-PSNR T	(%) (%) (dB) (%)	27.22 1.70 -0.017 -48.88	59.66 1.53 -0.059 -66.38	50.49 1.65 -0.052 -47.77	-2.79 -0.081 -70.82	53.06 1.73 -0.033 -53.85	51.89 1.75 -0.030 -65.62	49.16 2.02 -0.045 -52.77	46.55 1.67 -0.061 -48.23	43.32 1.38 -0.090 -48.07	30.33 0.96 -0.038 -59.30	27.08 1.47 -0.055 -54.32	38.24 1.26 -0.056 -56.67	34.30 1.27 -0.046 -60.79	39.87 0.97 -0.034 -50.14	-1.60 -0.018 -50.33	78.98 2.71 -0.071 -72.42	79.31 2.46 -0.083 -73.59	77.58 2.93 -0.094 -74.91	54.57 1.78 -0.053 -58.60
	.5: Comparative Study	[LZZ + 19] [TTAA19]	T BD-BR BD-PSNR T BD-BR BD-PSNR	(%) (%) (dB) (%) (%) (dB)	-47.50 5.45 -0.250 29.84 1.85 -0.085	-60.60 ---3.20 -0.102	-56.03 0.35 -0.010 34.74 2.43 -0.079	-58.72 2.84 -0.090 46.42 --	-56.87 2.79 -0.065 43.22 2.46 -0.060	-60.01 2.15 -0.038 38.70 1.74 -0.032	-55.84 2.06 -0.046 39.45 1.93 -0.046	-55.19 3.90 -0.148 32.12 2.24 -0.089	-50.74 5.56 -0.227 37.11 1.86 -0.071	-46.83 4.74 -0.210 31.58 2.58 -0.108	-46.22 2.10 -0.180 26.03 1.25 -0.048	-49.04 ---3.12 -0.147	-46.91 3.38 -0.145 35.72 3.02 -0.117	-45.62 3.41 -0.136 24.73 3.99 -0.154	-41.86 -----	-64.37 1.66 -0.058 65.28 1.83 -0.063	-66.49 0.90 -0.020 64.05 5.45 -0.139	-67.23 1.58 -0.050 64.67 3.30 -0.108	-54.2 2.56 -0.099 43.33 2.97 -0.107
		Sequence [XLW + 18]	BD-BR BD-PSNR	(%) (dB)	PeopleOnStreet 1.05 -0.045	Traffic 1.99 -0.052	Kimono 1.49 -0.048	ParkScene 1.47 -0.042	Cactus 2.07 -0.043	BQTerrace 1.09 -0.017	BasketballDrive 2.26 -0.052	BasketballDrill 1.95 -0.072	BQMall 1.91 -0.071	PartyScene 1.01 -0.039	RaceHorses 0.87 -0.032	BasketballPass 1.45 -0.066	BQSquare 0.77 -0.028	BlowingBubbles 1.29 -0.044	RaceHorses 1.11 -0.047	FourPeople 1.83 -0.052	Johnny 1.69 -0.038	KristenAndSara 1.55 -0.045	Overall 1.49 -0.046

DEEP LEARNING BASED VIDEO QUALITY ENHANCEMENT FOR THE NEW VERSATILE VIDEO CODING

Table III.3 shows the comparison of encoding performance with other approaches CHAPTER III.

Table IV

 IV

		.3: Hardware Resource Occupation of the FC-IP
	IP cores	Resource	BRAM_18k	DSP	FF	LUT
	FC1-IP	Total	4	5	1151	1618
		Available	280	220	106400
		Utilization (%)	1	2	1	3
	FC2-IP2	Total	1	5	1151	1618
		Available	280	220	106400
		Utilization (%)	0	2	1	3
	FC3-IP	Total	0	5	1149	1618
		Available	280	220	106400
		Utilization (%)	0	2	1	3

Table IV

 IV CNNs based on Xilinx VC709 board. This proposed method almost entirely uses the hardware resources of the FPGA, consuming almost all the DSPs, over 60% LUTs, 50%

	CHAPTER IV. DEEP CNN CO-DESIGN FOR HEVC CU PARTITION
	PREDICTION ON FPGA-SOC
		Table IV.5: Comparative Study
		[LCX + 19]		[ZWCL21]	[LZF + 19]	Our Design
	Platform	XC7VX690T	XC7Z035	7100		XC7Z020
	Frequency (MHz)	120		200	100		142
	LUTs	62.9%		48.4%	51		47%
	FFs	50.2%		31.7%	38		26%
	BRAMs	26.6%		74%	46		16%
	DSPs	99.8%		21.3%	95		14%
	Power (W)	15.8		5.96	3.99		1.69
			.4: Hardware Cost
	Resource	LUT	LUTRAM FF	BRAM DSP IO	BUFG
	Total	25026 1374	27979	22	30	4	1
	Available	53200 17400	106400 140	220	125 32
	Utilization (%)	47	8	26	16	14	3	3
	FPGA-Frequency (MHz) 142
	PS7-Frequency (MHz)	525
	Bitstream (Ko)	3951

II.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

(a) ClassA1 (b) ClassA2 (c) ClassB (d) ClassC (e) ClassD

Acknowledgments

List of

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY

computational complexity, the bi-threshold decision scheme was adopted at three levels.

Note that the upper and the lower thresholds at level l are represented by (γ l ) 3 l=1 and (γ l ) 3 l=1 . At three levels, the LSTM network provides the predicted CU partition probability P l (CU ). Consequently, the CU decides to be split only when P l (CU ) > γ l . If P l (CU ) < γl , the CU is not split. The interval [γ l , γ l ] represents the uncertain zone, in which the possible splitting patterns of the current CU need to be traversed by HEVC for the RDO search. In this way, the HEVC complexity is reduced considerably by skipping the most redundant verification of the RD cost.

II.4 Experimental Results

This section introduces the evaluation performance of the proposed HEVC method.

Specifically, we first present the experimental settings. Then, the performance metrics are provided. Finally, the evaluation performance of the proposed machine learning approaches are discussed and compared to other related algorithms.

II.4.1 Experimental Settings

In this section, we present the obtained results to validate the coding efficiency of the proposed deep learning framework. Our experiments were performed in the HM16.5 reference test model [Mod] using the Low Delay P (LDP) configuration. The QP values tested were 22, 27, 32 and 37 for encoding process. All simulations were tested on 18

JCT-VC videos from class A (2650×1600) to class E (1280×720) [B + 13], as illustrated in Table II.2. The frames number used for each video sequences is 100. All implementations were executed on windows 10 OS platform with Intel ®core TM i7-3770 @ 3.4 GHz CPU and 16 GB RAM. To accelerate the speed of the network model training phase, we also used the NVIDIA GeForce GTX 480 GPU, but it was not used in the HEVC complexity reduction test. In the experiments, the Tensorflow-GPU deep learning framework was used.

CHAPTER II. MACHINE LEARNING APPROACH-BASED FAST CU PARTITION FOR REDUCING HEVC COMPLEXITY

On the other hand, the proposed approach can reduce the BD-PSNR performance by -0.053dB, which is better than -0.057dB achieved by the proposed Deep CNN. Furthermore, our proposed approach has an average BD-BR performance of 1.78%, better than that of [START_REF] Bouaafia | Fast cu partition-based machine learning approach for reducing hevc complexity[END_REF]; 1.80%. In our experiments, we note that the proposed deep learning CNN-LSTM achieves high HEVC complexity reduction at inter-coding, because it is capable to predict all the CU splitting of an entire CTU at the same time. The proposed algorithm also performs well in terms of BD-PSNR performance, due to the high accuracy of the predicted CU partition. Consequently, the learning scheme based on CNN-LSTM achieves a good compromise between RD performance and coding complexity in order to predict inter-mode CU partition of HEVC. This is mainly due to the LSTM ability to resolve the temporal correlation through adjacent frames.

For more evaluation, the reducing complexity of CNN-LSTM versus deep CNN under all video sequences (A -E) at LDP configuration is proved in Figure II.9. As shown in this Figure, the proposed approach allows higher encoding time when the QP value increases from 22 to 37. Overall, the proposed deep learning approach outperforms best in terms of time saving than the deep CNN. Consequently, the proposed scheme is better for reducing the HEVC complexity of inter-coding and for finding an optimal CU partition, compared to traditional RDO research.

II.4.5 Comparative Study

To evaluate the encoding performance of the proposed learning approach, our experimental results are compared to other state of the art methods. The CTU level on/off control is adopted to avoid a reduction in RDO performance.

The frame level filtering would be shut off to prevent over-signal, if the enhancement quality is not worth to cost the signaled bits. Specifically, the control flags at the CTU-

III.5 Experimental Results

In this section, we will present the performance of the proposed loop filtering scheme based on WSE-DCNN in the VVC standard through experimental results. We evaluated the performances, in terms of RD performance, and QoE of the proposed scheme. Specifically, we first present the description of the dataset collection in the experiments. Then, model training, testing, and evaluation are provided. Finally, objective evaluations and subjective visualizations are presented. Furthermore, the performance of the proposed method is illustrated and compared with other CNN based in-loop filter algorithms.

III.5.1 Dataset Collection

In

III.5.3 WSE-DCNN Evaluation

The RD performance results of the proposed model compared to the original VVC standard are shown in Table III

The BQSquare video sequence encoded with QP equals to 37 under RA configuration is deployed in order to show the subjective visual quality and to further verify the effec-

IV.2.1 FPGA-SoC

Nowadays, the limitations of Application Specific Integrated Circuit (ASIC) Systemon-Chips (SoCs) make them incompatible with a large number of applications in terms of time-to-market, flexibility and upgrade capability. There is a clear need for a more flexible solution, and this is what motivates the System-Programmable-Chip, a specific flavour of SoC implemented on a programmable, reconfigurable device. However, the FPGAs is the popular solution used, which are inherently flexible devices that can be configured to implement any arbitrary system, including embedded processors if needed.

FPGAs can also be reconfigured as often as desired, thus offering a more fundamentally flexible platform than ASICs for implementing SoCs.

IV.2.1.3 PS-PL Interfaces

In this section, we introduce the connections between the PS and PL and consider how they can be used. We start by introducing the AXI standard, on which most of these connections are adopted. Suite. AXI buses can be used flexibly, and in general are used to connect the processor to other IP blocks in an embedded system. In fact, there are three versions of AXI4, each representing a different bus protocol, as summarised below. The choice of AXI bus protocol for a particular connection depends on the desired properties of that connection.

In fact, AXI4 is for memory-mapped links and providing the highest performance; an address is supplied followed by a data burst transfer of up to 256 data words. AXI4-Lite is a simplified link supporting only one data transfer per connection. AXI4-Lite is also memory-mapped; in this case an address and single data word are transferred.

Then, AXI4-Stream is for high-speed streaming data and supporting burst transfers of unrestricted size. There is no address mechanism; this bus type is best suited to direct data flow between source and destination (non memory mapped). In addition,

IV.2.4 High Level Synthesis (HLS)

Vivado High Level Synthesis (HLS) allows functions written in C, C++, SystemC and OpenCL kernels to be synthesized into a Register-Transfer-Level (RTL) implementation [START_REF] Vh Xilinx | Vivado design suite user guide-high-level synthesis[END_REF]. Vivado HLS provides a number of optional C libraries to enable higher productivity and high performance RTL design. These include arbitrary precision libraries allowing operations to be performed at any arbitrary precision.

IV.3.1 Proposed Deep CNN Architecture

As mentioned in the previous chapter, the HEVC standard adopts a quadtree structure, known as CTU. CTU supports CU partition from 64 × 64 to 8 × 8 at four levels.

The CU quadtree partition consumes much of the HEVC encoding complexity, due to the adoption of a wide variety of CU sizes at the RDO level. Therefore, the cost of computational complexity remains a critical issue that must be properly considered in The proposed CNN has better performance compared to conventional techniques, but it requires higher computational complexity which is due to the large number of parameters in convolutional and fully connected layers. To solve this issue, this model will be accelerated on a hardware platform.

IV.3.2 CNN Accelerator based on Vivado HLS

In this chapter, we have chosen to accelerate the first CNN model (CNN_1) corresponding to level 1 for a 64 × 64 CU partition. In the proposed deep CNN_1 model, three convolutional layers are used, each one is characterized by specific parameters such as the filters number and the kernel size, as mentioned in Table IV.1. All the processed data pass through the CONV operation to extract the feature maps in a 16 × 16 format. For CONV_2, the input image sized to 16 × 16, is convoluted with 2 × 2 kernels to generate the 8 × 8 output data. Additionally, for CONV_3, the input image sized to 8 × 8 is convoluted with 2 × 2 kernels to extract the 4 × 4 output data.

Following the above acceleration steps mentioned in the section IV.2.4, each CONV layer is accelerated using Vivado HLS tool and then exported as an RTL core (CONV1_0, CONV2_0, and CONV3_0). Therefore, these generated IP cores will be integrated and used in the Vivado design.

On the other hand, the proposed CNN_1 model contains three fully connected layers with different parameters, as shown in Table IV.1. We accelerate these three fully Therefore, these generated IP cores will be integrated as a new repository to be used in the design.

IV.3.3 Hardware-Software Co-Design for CNN on FPGA-SoC

Currently, there are two implementation CNN modes due to its hierarchical structure.

The first one is the Streaming Architectures and the second is the Single Computation Engine. The former has the ability to allocate corresponding hardware resources to each (network layer) IP core (CONV-IPs and FC-IPs) and it has the following characteristics. Firstly, it can realize the inter-layer pipeline and flexible control and management within each IP core with a high customization degree. Secondly, it can be applied only to small network layers (minimum neurons by each layer), since, it is characterized by its higher demand for resources. The latter indicates that different network layers share the same accelerator (one CNN-IP) through resource reuse, which is a non-highly customized architecture, is more inflexible but it is easier. Considering all these advantages, we buffer uses the AXI-stream interface to receive data and weights from the DMA. After the processing task by the accelerator, this IP core will provide outputs that will be sent back to the DDR via the stream_output and AXI-DMA. Thus, this output data, from the current IP core, will be the input for the second layer (second IP core) with the corresponding weight and parameters. That is why each IP core in the co-design includes tow data exchanges which are the input_stream and the output_stream within the AXI-lite for the parameter configurations and the stream_kernel for kernel's values programming (only for CONV-IP cores). The above operation will be repeated until the data processing of the network model is completed. On the other hand, the PS reset core, within the AXI-gpio interfaces, generates a customized reset for the entire system, such as the AXI-Interconnect and peripherals.

CHAPTER IV. DEEP CNN CO-DESIGN FOR HEVC CU PARTITION PREDICTION ON FPGA-SOC

IV.4 Experimental Results

This section introduces the performance evaluation of the hardware accelerators and the full co-design system. The provided results are implemented on Vivado pack v2016.1 within PYNQ-Z1 board (Xilinx Zynq-7000 device).

IV.4.1 IP Cores Hardware Resource

The three CONV-IP hardware resource occupation of the target FPGA are summarized in Table IV.2. The hardware CONV1-IP core occupies 3% of BRAMs to store the parameters, 3% of LUTs, 2% of DSP, while using only 1% of FFs. In addition, the CONV2-IP uses 1%, 2% and 4% of DSPs, FFs, and LUTs, respectively. For CONV3-IP core, the resources used are 2% of DSP, 1% of FFs, and 4% of LUTs. To sum up, the CONV1-IP presents 9% of hardware cost with a frequency of 150 MHz. While the CONV2-IP and CONV3-IP occupies 7% of hardware cost on the PYNQ-Z1 with a frequency of 150 MHz also. This is considered a low occupancy on the PYNQ-Z1 FPGA with high processing speed. According to the power report, the static power of our proposed system is about 0.160W and the dynamic power is 1.539W . However, the total on-chip power is 1.699W of the proposed architecture.

IV.4.3 Comparative Study

The proposed design has also been compared to other related works.

List of Publications

Appendix A

Convolutional Neural Networks

CNNs have achieved remarkable success in the fields of image processing and computer vision [START_REF] Bouaafia | Fast cu partition-based machine learning approach for reducing hevc complexity[END_REF][START_REF] Bouaafia | Cnn-lstm learning approach-based complexity reduction for highefficiency video coding standard[END_REF]. Indeed, the key terminology and operations involved in CNNs architecture, including convolution, normalization, pooling, activation functions, and fully connected layers have been introduced [START_REF] Kumar | Recent deep learning techniques, challenges and its applications for medical healthcare system: A review[END_REF].

A.1 Convolutional Layer

Convolutional Layers are the basic building blocks, which are the most computationally

A.2 Activation Functions

The commonly used activation function is the Rectified Linear Unit (ReLU) which clips all negative values to zero. It is a non-linear activation function used after the convolutional layer, which can be defined as.

where the weighted sum of the neuron inputs denoted by x. The goal of ReLU is to converge faster in training and has low computational complexity compared to other functions. Besides, sigmoid and tanh functions are another popular activation function, which can be given as.

A.5 Backpropagation Algorithm

The backpropagation algorithm is resumed using four equations. In order to see this, we must first establish some useful notation. We will assume that there are L layers The four backpropagation equations are defined by the equations A.8, A.9, A.10, and A.11, relating the gradients of the activations of various neurons a l j , the weighted inputs z l j in A.6, and the errors ∆ l j . This algorithm consists of a forward pass from the bottom layer to the top layer where one calculates the weighted inputs and activations of all the neurons. One then backpropagates the error starting with the top layer down to the input layer and uses these errors to calculate the desired gradients. This description makes clear the incredible utility and computational efficiency of the backpropagation algorithm. We can calculate all the derivatives using a single "forward" and "backward" pass of the neural network. This computational efficiency is crucial since we must calculate the gradient with respect to all parameters of the neural network at each step of gradient descent.

116