• Reducing Anomaly Detection in Images to Detection in Noise, Axel

This thesis focuses on denoising and in particular video denoising. For a long time, it was not possible to envisage sophisticated video denoising. This is due to constraints linked with video processing. While it is possible for an image to require up to a minute of processing, or even processing it remotely, this is not possible for a video because of the large number of frames that makes a even a short video. Even though the problem is very similar to image denoising, video denoising requires specific methods. We show here that not only using the temporal information in video is very important but also that efficient implementations on GPU allow for real time video denoising without any major modification of the methods. We also show that the mechanisms developed can be used for other applications such as anomaly detection and forgery detection.

The first part of the thesis focuses on patch-based methods. We first study one of the most popular and successful video denoising method called VBM3D. It combines self-similarity, by looking for similar patches both spatially and temporally, as well as patch filtering. We then take a more in-depth look at the patch search step for these patch-based methods. In particular we show the importance of temporal patch search in video. A patch search that searches in the entire video allows for better denoising. We also propose a novel causal and recursive video denoising method called NL-Kalman that tries to bridge the gap between lighter temporal filtering methods and heavier non-causal self-similar methods. Finally, we also show that patch-based methods can be implemented efficiently of GPU allowing for real time video denoising without any loss in quality.

We investigate the new trend of machine learning methods for denoising and low-level processing in the second part. We first propose one of the first neural network architecture for video denoising that can compete with the state of the art. In particular we introduce temporal information into the network using self-similarity similarly to patch-based methods. We also show that deep learning offers new opportunities. In particular, it allows for denoising without knowing the noise model. Thanks to the proposed framework, videos that have been processed with an unknown processing pipeline can still be denoised. Finally, we take a look at RAW mosaicked data. Indeed demosaicking and denoising are usually the two first steps of any image processing pipeline. Some deep learning methods suggest solving the two problems at the same time. We review the current state of the art for image demosaicking and show that deep learning methods are much better both qualitatively and quantitatively than hand-crafted methods. We also show that such demosaicking methods can be trained without requiring ground truth by simply using pairs of mosaicked images of the same scene. This process allows for realistic joint demosaicking and denoising and even single burst fine-tuning.

The last part of this thesis apply mechanisms used for denoising methods to other applications. In particular, we first look at the problem of anomaly detection in images. We show that this problem can be reduced to detecting anomalies in residual images in which noise and anomalies prevail. We also look at other applications of patch matching such as forgery detection with the detection of copy-paste forgeries. We analyze the theory of the the famous PatchMatch method before presenting how it can be used for copy-paste detection. We also apply an a contrario patch matching method to increase the robustness to similar objects.

Je tiens tout d'abord à remercier mes directeurs de thèse (Jean-Michel Morel, Pablo Arias et Gabriele Facciolo) pour l'opportunité ainsi que leur aide durant la préparation de cette thèse.

Résumé

Cette thèse se concentre sur le débruitage et en particulier le débruitage vidéo qui a pendant longtemps été ignoré. La raison principale est les contraintes liées au traitement de la vidéo. Pour une image, il est acceptable d'avoir besoin de plusieurs minutes de traitement, voire même de devoir le faire sur un serveur distant. Cela n'est pas acceptable pour une vidéo en raison de la taille des données. Bien que le débruitage d'image et de vidéo sont des problèmes très similaires, il est nécessaire d'avoir des outils spécifiques adaptés à la vidéo. Nous montrons ici que l'information temporelle est très importante pour le traitement de la vidéo mais aussi que des implémentations efficaces sur carte graphique permettent d'avoir des méthodes temps-réel sans avoir à les modifier. Nous montrons aussi que les outils utilisés pour le débruitage peuvent être utilisés pour d'autres applications comme la détection d'anomalies ou la détection de falsification.

La première partie de cette thèse est dédiée aux méthodes à patches. Nous étudions dans un premier temps VBM3D, l'une des méthodes les plus populaires. Elle combine l'auto-similarité, en cherchant des patches similaires à la fois spatialement et temporellement, et le filtrage des patches. Nous étudions en détail l'étape de recherche de patches des méthodes à patches. Nous montrons en particulier l'importance de la recherche temporelle de patches dans la vidéo. En effet, une recherche plus globale permet d'avoir de meilleures performances de débruitage. Nous proposons aussi une nouvelle méthode de débruitage, appelée NL-Kalman, qui est à la fois récursive et causale. Avec NL-Kalman, nous essayons de réduire l'écart entre les méthodes de type filtrage temporelle qui sont légères et les méthodes non-causales basées sur l'auto-similarité qui, bien que très efficaces, sont beaucoup plus coûteuses. Enfin, nous montrons que les méthodes à patches peuvent être implémentées efficacement sur carte graphique. Cela permet d'avoir un débruitage temps-réel sans compromis sur la qualité.

Dans une deuxième partie, nous étudions une nouvelle tendance pour le débruitage et les traitements bas-niveaux basée sur les méthodes d'apprentissage. Nous proposons tout d'abord l'une des premières architectures de réseaux de neurones pour le débruitage vidéo qui est compétitif avec l'état de l'art. Pour cela nous introduisons de l'information temporelle dans le réseau grâce à l'auto-similarité, comme dans les méthodes à patches. Nous montrons aussi que les méthodes basées sur l'apprentissage profond offrent de nouvelles opportunités. En particulier, il devient possible de débruiter sans connaître le modèle du bruit. Grâce à la méthode proposée, même les vidéos traitées par une chaîne de traitement inconnue peuvent être débruitées. Nous considérons aussi des données brutes encore mosaïquées. En effet, le démosaïquage et le débruitage sont, en général, les deux premières étapes d'une chaîne de traitement d'images. Certaines méthodes d'apprentissage proposent de résoudre les deux problèmes en même temps. Après avoir examiné l'état de l'art du démosaïquage, nous pouvons conclure que les méthodes basées sur les réseaux de neurones sont meilleures de façon quantitative mais aussi visuel que les autres méthodes. Nous montrons aussi que ces méthodes peuvent être entraînées sans vérité terrain en utilisant uniquement des paires d'images mosaïquées de la même scène. Ce procédé permet d'apprendre un démosaïquage et débruitage réaliste et même d'adapter l'apprentissage à une acquisition rafale spécifique.

La dernière partie de cette thèse concerne les applications à d'autres problèmes des mécanismes utilisés pour le débruitage. L'un de ces problèmes est la détection d'anomalies dans les images. Nous montrons que ce problème peut être ramené à détecter des anomalies dans une image résiduelle dans laquelle du bruit et les anomalies dominent. Nous étudions d'autres applications à la recherche de patches comme la détection de falsifications telles que la détection de copié-collé. Nous analysons la théorie derrière la célèbre méthode PatchMatch avant de présenter comment elle peut être utilisée pour la détection de copié-collé. Nous utilisons aussi la comparaison de patches a contrario pour augmenter la robustesse aux objets similaires.

12 Detection of copy-paste forgeries based on sparse descriptors 12.

Introduction

Motivations

Denoising is a fundamental image and video processing problem. While the performance of denoising methods and imaging sensors has steadily improved over decades of research, new challenges have also appeared. High-end cameras still acquire noisy images in low lighting conditions. High-speed video cameras use short exposure times, reducing the SNR of the captured frames. Cheaper, lower quality sensors are used extensively, for example in mobile phones or surveillance cameras, and require denoising even with a good scene illumination. This state of facts is amplified by the poor quality of the optic used for these devices, due to cost and space constraints.

Its relevance has increased with the democratization of mobile imaging devices such as smartphones due to the wide variety of usage. For example, these small devices are often used in poor lighting condition. This degrades the quality of the output image, and the dominant factor in that quality loss is noise. An example of the necessity of denoising even for high-end modern smartphones is shown in Figure 1.

Figure 1: While the final output of a Samsung Galaxy S7 doesn't show apparent noise (left), the RAW image seen by the sensor is actually very noisy (right). The final output is obtained thanks to a complex processing that applies denoising as one of its step.

The same problems also arise with other image sensing devices. In particular the same trend of smaller and cheaper sensors acquiring many images quickly can be observed in remote sensing imaging. For example, one of the largest provider of satellite images Planet can now acquire short videos, also called bursts. While these images have a worst ground resolution and more noise, multi-frame denoising and super-resolution [MSS + 14] can be applied to create a single higher-quality image. Denoising has also been applied to non-optical imaging devices such as radar images [DDT + 14], ultrasound [START_REF] Coupé | Nonlocal means-based speckle filtering for ultrasound images[END_REF], fMRI images [START_REF] Meije | Denoising functional mr images: a comparison of wavelet denoising and gaussian smoothing[END_REF] and fluorescent images [BKB + 09] for medical images.

An additional problem with modern devices is the prevalence of proprietary pipelines. Images are available at the end of an usually unknown processing pipeline or, sometimes, completely raw data. The noise at the end of the pipeline is difficult to model and quite different from the popular Gaussian noise model assumed in many methods. While the noise of raw data is easy to model, a difficulty is caused by mosaicked data. Indeed, most cameras capture only one color per pixel, this color being determined by a color filter array (CFA) located on top of the sensor. Such data requires additional processing steps. In this thesis we try to take into account all practical challenges that come with modern denoising by looking at the possibilities offered by model-blind denoising and also looking at joint demosaicking and denoising.

While denoising is the main focus of this thesis, we also take a look at the problem of anomaly detection. The automatic detection of anomalous structure in arbitrary images is concerned with the problem of delineating regions not conforming with the rest of the data. For example, this is particularly important to detect defects in a production line. We shall put in evidence a striking similarity between methods for anomaly detection and for denoising. We also take a look at forgery detection, another problem with similarity to both denoising and anomaly detection.

Part I (Chapters 1 to 4) presents and extends traditional patch-based methods. In Part II (Chapters 5 to 8), we take a look at the more recent trend of machine learning for low-level vision processing and the new exciting opportunities they bring to the table. Finally, Part III (Chapters 9 to 12) deals with detection applications using similar tools than the ones developed for denoising.

VBM3D and its extensions

In this chapter we review one of the most popular video denoising methods. VBM3D is an extension to video of the well-known image denoising algorithm BM3D, which takes advantage of the sparse representation of stacks of similar patches in a transform domain. The extension is rather straightforward: the similar 2D patches are taken from a spatio-temporal neighborhood which includes neighboring frames. In spite of its simplicity, the algorithm offers a good trade-off between denoising performance and computational complexity. A detailed description is given and the choice of parameters is thoroughly discussed. Furthermore, we study and compare several extensions of the original algorithm: (1) a multi-scale implementation, (2) the use of 3D patches, (3) the use of optical flow to guide the patch search. These extensions allow to obtain results which are competitive with even the most recent state of the art. Figure 2 presents the core structure of VBM3D. This method will then be used as reference to compare the denoising methods proposed in the following chapters. Understanding in details all the mechanisms of the method also led to an optimized GPU implementation presented in Chapter 4.

Global patch-search for denoising

As we show in Chapter 1, patch search is an important mechanism of patch-based methods, and it will be recurrent throughout the thesis. In the case of a single image, a local search region is justified by the fact that similar patches are likely to be close to each other in the image domain. Videos however, have an additional strong source of redundancy given by the temporal consistency. A patch is expected to have similar patches along its motion trajectory, even in distant frames. It seems intuitive that patch-based methods should benefit from this larger set of similar exemplars. However, this search is generally performed locally around each target patch for obvious complexity reasons.

In this chapter we focus on the patch search. We present an efficient global approximate search technique and demonstrate its impact on video denoising. It permits for the first time to evaluate the impact of a global search on the video denoising performance.

Figure 3 shows that the method achieves a behavior similar to a global exhaustive search, for a fraction of the computational cost.

A recursive video denoising method: NL-Kalman

Contrary to the methods presented in Chapters 1 and 2, we propose a recursive causal method in this chapter. This means that the method uses only the current frame and the previous denoised one. The goal of such methods is to close the gap between high-quality but slow methods and those efficient but of lower quality.

The proposed method, called NL-Kalman, considers the video as a set of overlapping temporal patch trajectories. Following a Bayesian approach each trajectory is modeled as a linear dynamic Gaussian model and denoised by a Kalman filter. To estimate its parameters, similar patches are grouped and their trajectories are considered as sharing the same model parameters. The filtering is mainly temporal; non-local spatial similarity is only used to estimate the parameters. This temporally causal method obtains results comparable (in terms of PSNR and SSIM) to state-of-the-art methods using several frames per frame denoised but with a better temporal consistency. While temporal consistency cannot be visualized without looking at the video, Figure 4 Figure 4: Denoising results when the sequence is corrupted by noise of standard deviation 30. From left to right: Denoising using NL-Kalman, NL-Kalman with an oracle optical flow, VBM3D, VBM4D and the original frame. From top to bottom: Crop from the sequence crowd_run and pedestrian_area. Overall the proposed method preserves more details as it can be seen both in the tree and in the text. Results best viewed zoomed. shows that the proposed method also recovers more details.

Method

Fast patch-based denoising on GPU

We show in the previous chapters that denoising is an essential part of any image or video processing pipeline. Unfortunately, due to time processing constraints, many pipelines do not consider the use of modern denoisers. These algorithms have only CPU implementations or suboptimal GPU implementations. In this chapter, we propose a new efficient GPU implementation of NLmeans and BM3D, and, to the best of our knowledge, the first GPU implementation of the video denoising algorithm VBM3D. The performance of these implementations enable their use in realtime scenarios. Table 1 shows the impact of an optimized implementation for video denoising.

Non-Local video denoising by CNN

Non-local patch based methods were until recently state of the art for image denoising but are now outperformed by machine learning and in particular CNNs. Yet they are still the state of the art for video denoising, as video redundancy is a key factor to attain high denoising performance. The problem is that CNN architectures are hardly compatible with the search for self-similarities. In this chapter we propose a new and efficient way to feed video self-similarities to a CNN. The non-locality is incorporated into the network via a first non-trainable layer which finds for each patch in the input image its most similar patches in a search region. The central values of these patches are then gathered in a feature vector which is assigned to each image pixel. This information is presented to a CNN which is trained to predict the clean image. We apply the proposed architecture to image and video denoising. For video, patches are searched for in a 3D spatio-temporal volume. The proposed architecture, shown in Figure 5, achieves state-of-the-art results.

Model-blind video denoising via frame-to-frame training

In the previous chapters, the noise model is always considered fixed and known. However, modeling the processing chain that has produced a video is a difficult reverse engineering task, even when the camera is available. This makes model based video processing a still more complex task. In this chapter we propose a fully blind video denoising method, with two versions offline and online. This is achieved by fine-tuning a pre-trained AWGN denoising network to the video with a novel frame-to-frame training strategy. Our denoiser can be used without knowledge of the origin of the video and the post-processing steps applied from the camera sensor. The online process only requires a couple of frames before achieving visually pleasing results for a wide range of perturbations. It nonetheless reaches state-of-the-art performance for standard Gaussian noise, and can be used offline with still better performance. The flexibility of the proposed method is shown in Figure 6.

Demosaicking with deep learning

Most cameras capture the information of only one color for a given pixel. This results in a mosaicked image that must be interpolated to get three colors at each pixel. The step going from a mosaicked image to a regular RGB image is called demosaicking. This chapter reviews two recent demosaicking methods based on convolutional neural networks that achieve artifactfree state-of-the-art results: Deep joint demosaicking and denoising by Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] and Color image demosaicking via deep residual learning by Tan et al. [START_REF] Tan | Color image demosaicking via deep residual learning[END_REF]. These methods beat by almost two decibels the best human-crafted methods, while being faster by one order of magnitude. A difficult example of demosaicking where only learning methods performs well is shown in Figure 7. This, arguably, seals the destiny of human-crafted methods on this subject. , MLRI [START_REF] Kiku | Minimized-laplacian residual interpolation for color image demosaicking[END_REF], ARI [START_REF] Monno | Adaptive residual interpolation for color image demosaicking[END_REF], Getreuer [START_REF] Getreuer | Image Demosaicking with Contour Stencils[END_REF]. Learning based methods produce fewer artifacts while still being very efficient computationally.

Figure 8: Using a burst, our fine-tuning (starting from the network from Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF]) is able to not only denoise well (σ = 5) but also doesn't show any artifacts like zipper or moire in the difficult regions. Best visualized on a screen.

Joint demosaicking and denoising by fine-tuning of bursts of raw images

Demosaicking, studied in Chapter 7, and denoising are the first steps of any camera image processing pipeline and are key for obtaining high quality RGB images. A promising current research trend aims at solving these two problems jointly using convolutional neural networks. Due to the unavailability of ground truth data these networks cannot be currently trained using real RAW images. Instead, they resort to simulated data.

In this chapter we present a method to learn demosaicking directly from mosaicked images, without requiring ground truth RGB data. We apply this to learn joint demosaicking and denoising only from RAW images, thus enabling the use of real data. This allows to mitigate the training bias due to simulated data. In addition we show that for this application fine-tuning a network to a specific burst improves the quality of restoration for both demosaicking and denoising. An example on synthetic data is shown in Figure 8.

How to Reduce Anomaly Detection in Images to Anomaly Detection in Noise

While denoising is important to improve the visual quality of images and videos, it can also have a surprising application to seemingly completely unrelated problems. One of such surprising applications is detecting anomalies. Anomaly detectors address the difficult problem of detecting automatically exceptions in a background image, that can be as diverse as a fabric or a mammography. Detection methods have been proposed by the thousands because each problem requires a different background model. By analyzing the existing approaches, we show in this chapter that the problem can be reduced to detecting anomalies in residual images (extracted from the target image) in which noise and anomalies prevail. Hence, the general and impossible background modeling problem is replaced by a simple noise model and allows the calculation of rigorous detection thresholds. Our approach is therefore unsupervised and works on arbitrary images. The residual images can favorably be computed on dense features of neural networks. Our detector is powered by the a contrario detection theory, which avoids over-detection by fixing detection thresholds taking into account the multiple tests. Figure 9 shows detections on real natural images using different methods for anomaly detection. Figure 9: Detection results of our method when using as image representation directly the image pixels or the activation maps of the VGG neural Network at different layers (conv1_1, conv2_1, conv3_1) and a comparison to [START_REF] Mishne | Multiscale anomaly detection using diffusion maps[END_REF], [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF], [JWY + 13] and [START_REF] Huang | Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks[END_REF] on real examples. The first two images (top and second row) are part of the Toronto dataset [START_REF] Bruce | Saliency based on information maximization[END_REF], while the third and fourth row are from [START_REF] Mishne | Multiscale anomaly detection using diffusion maps[END_REF] and [START_REF] Tsai | Automated surface inspection for directional textures[END_REF] respectively.

Analysis of PatchMatch

As seen in the previous chapters, many problems in image/video processing and computer vision require the computation of a dense k-nearest neighbor field (k-NNF) between two images. For each patch in a query image, the k-NNF determines the positions of the k most similar patches in a database image. With the introduction of the PatchMatch algorithm [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF], Barnes et al. demonstrated that this large search problem can be approximated efficiently by collaborative search methods that exploit the local coherency of image patches. After its introduction, several variants of the original PatchMatch algorithm have been proposed, some of them reducing the computational time by two orders of magnitude. In this chapter we study the convergence of PatchMatch and its variants, and derive bounds on their convergence rate. We consider a generic PatchMatch algorithm from which most specific instances found in the literature can be derived as particular cases. We also derive more specific bounds for two of these particular cases: the original PatchMatch and Coherency Sensitive Hashing [START_REF] Korman | Coherency sensitive hashing[END_REF]. The proposed bounds are validated by contrasting them to the convergence observed in practice.

Detection of copy-paste forgeries based on PatchMatch

This chapter reviews an application of the PatchMach process studied in Chapter 10. The method studied in this chapter has been presented in [START_REF] Cozzolino | Efficient dense-field copy-move forgery detection[END_REF]. This method is a forgery detection based on a dense field of descriptors chosen to be invariant by rotation, for example Zernike moments.

An efficient matching of the descriptors is then performed using PatchMatch, which is extremely efficient to find duplicate regions. Regions matched by PatchMatch are processed to find the final detections. This allows a precise and accurate detection of copy-move forgeries inside a single suspicious image. We also extend successfully the method to the use of dense SIFT descriptors and show that they are better at detecting forgeries using Poisson editing. An example of detection is shown in Figure 10. 12. Detection of copy-paste forgeries based on sparse descriptors

The method presented in Chapter 11 suffers from over-detecting similar but genuinely different regions. This is because detecting reliably copy-move forgeries is difficult and in particular because images do contain similar objects. These similar objects are detected because most methods are not robust enough to differentiate clones and similar objects. The question then is: how to discard natural image self-similarities while still detecting copy-moved parts as being "unnaturally similar"? Especially knowing that copy-move may have been performed after a rotation, a change of scale and followed by JPEG compression or the addition of noise.

For this reason, we propose another method based on robust patch matching similar to those presented in Part I. We use keypoints similar to SIFT, meaning sparse keypoints with scale, rotation and illumination invariant descriptors. To discriminate natural descriptor matches from artificial ones, we introduce an a contrario patch comparison method which gives theoretical guarantees on the number of false alarms. We validate our method on several databases. Being fully unsupervised it can be integrated into any generic automated image tampering detection pipeline. In Figure 11, we show how it is possible to distinguish similar but genuine objects and forgeries using only local image patches.

Publications

This thesis has led to the following publications: On the left image the two objects are similar but not digital copies of each other. On the right, one is a digital copy of the other. The patches shown below each respective image correspond to the red squares in the images. They show that a difference is visible at this level and therefore the descriptors can be discriminated. This is why the detection method can discard genuinely similar objects.

Introduction en français

Motivations

Le débruitage est un problème fondamental du traitement de l'image et de la vidéo. Bien que les performances des méthodes et des capteurs aient continué de s'améliorer au cours de dizaines d'années de recherche, de nouveaux défis sont apparus. Les appareils haut de gamme produisent encore des images bruitées dans des conditions de lumière difficiles. Les caméras ultra-rapide ont des temps de captures très courts, dégradant ainsi le rapport signal à bruit dans les images. Des capteurs moins chers et de moins bonne qualité sont utilisés à grande échelle pour les téléphones portable et les caméras de surveillance. Ces capteurs produisent des images bruitées même dans de bonnes conditions d'illumination. Ce problème est d'autant plus important que l'optique utilisée pour ces appareils est en général de mauvaise qualité à cause de leur coût et des contraintes de place. Son importance est d'autant plus grande avec la démocratisation des téléphones portables à cause de leurs utilisations très variées. Par exemple, ces appareils sont souvent utilisés dans des environnements peu éclairés. Cela dégrade la qualité de l'image produite et le facteur prédominant de cette perte de qualité est le bruit. Un exemple de l'importance du débruitage même avec un téléphone haut de gamme récents est montré dans la Figure 12. Les mêmes problèmes sont aussi présents avec d'autres systèmes d'acquisition d'images. En particulier, la tendance d'utiliser des capteurs plus petits et bon marché est aussi présente dans l'imagerie satellitaire. Par exemple, un des plus gros fournisseur d'images satellitaires, Planet, peut maintenant acquérir de courtes séquences d'images. Bien que ces images aient une moins bonne résolution spatiale et plus de bruit, du débruitage et de la super-résolution multi-image permettent de créer une unique image de bien meilleure qualité. Le débruitage a aussi été utilisé pour d'autres types d'imagerie comme des images radar [DDT + 14], ultrason [START_REF] Coupé | Nonlocal means-based speckle filtering for ultrasound images[END_REF], IRM [START_REF] Meije | Denoising functional mr images: a comparison of wavelet denoising and gaussian smoothing[END_REF] et de l'imagerie par fluorescence [BKB + 09] pour des images médicales.

Un problème supplémentaire avec les appareils modernes est l'existence de chaînes de traitement propriétaires. Les images sont soient disponibles après une chaîne de traitement inconnue, soient, dans certains cas, complètement brutes. Le bruit après la chaîne de traitement est difficile à modéliser et peut être très différent d'un bruit blanc gaussien qui est très souvent le modèle supposé par les méthodes. Bien que le bruit des images brutes est plus facile à modéliser, leur problème vient du mosaïquage des données. En effet, la très grande majorité des appareils ne capture qu'une couleur par pixel, cette couleur étant déterminée par le filtre de couleur (CFA) situé sur le capteur. Ces données nécessitent du coup des traitements supplémentaires. Dans cette thèse, nous essayons de prendre en compte tous les défis pratiques du débruitage "moderne" en étudiant le débruitage dit "aveugle", sans avoir besoin de connaître le modèle de bruit à l'avance, et la résolution jointe du débruitage et du démosaïquage.

Bien que le débruitage soit le sujet principal de cette thèse, nous considérons aussi le problème de détection d'anomalies. La détection automatique de structures anomaliques dans des images arbitraires a pour but de délimiter les régions non conformes avec le reste des données. Par exemple, il est particulièrement important de détecter les défauts d'un produit dans une chaîne de production. Nous mettrons en lumière une ressemblance frappante entre la détection d'anomalies et le débruitage. Nous étudierons aussi la détection de falsification, un autre problème avec de grandes similarités avec le débruitage et la détection d'anomalies.

La Partie I (Chapitres 1 à 4) présente et étend les méthodes à patches pour le débruitage vidéo. Dans la Partie II (Chapitres 5 à 8), nous considérons le nouvel axe prometteur de recherche qui propose d'utiliser des méthodes d'apprentissage pour résoudre les problèmes de traitement d'images et vidéos et les nouvelles opportunités que ces méthodes offrent. Pour finir, la Partie III (Chapitres 9 à 12) présente différentes applications de détection qui utilisent des outils similaires à ceux présentés pour le débruitage.

VBM3D et ses extensions

Dans ce chapitre nous étudions l'une des méthodes les plus populaires pour le débruitage vidéo. VBM3D est une extension à la vidéo du célèbre algorithme de débruitage d'image BM3D qui utilise une représentation parcimonieuse des groupes de patches similaires après transformation. L'extension est assez directe: les patches 2D similaires sont trouvés dans un voisinage spatiotemporel qui inclut des images vidéo voisines. Malgré sa simplicité, l'algorithme offre un bon compromis entre qualité de débruitage et temps de calcul. Une description détaillée de la méthode est donnée et le choix des paramètres discuté. Nous étudions aussi différentes extensions possible à la méthode originale: (1) une implémentation multi-échelle, (2) l'utilisation de patches 3D, (3) l'utilisation du flot optique pour guider la recherche de patches. Ces extensions permettent d'obtenir des résultats compétitif avec l'état de l'art. La Figure 13 présente le principe de fonctionnement de VBM3D.

Cette méthode est utilisée ensuite comme référence pour comparer les différentes méthodes proposées dans les chapitres suivants. La bonne compréhension de tous les mécanismes de la méthode a permis d'implémenter efficacement la méthode sur carte graphique. Cette implémentation est présentée dans le Chapitre 4.

Recherche de patches globale pour le débruitage

Comme nous l'avons montré dans le Chapitre 1, la recherche de patches est un mécanisme important des méthodes à patches. Ce mécanisme est récurrent tout au long de cette thèse. Dans le cas d'une image unique, chercher dans une région locale est justifié par le fait que les patches les plus similaires sont plus susceptibles de se retrouver proche dans l'image. Les vidéos ont cependant une autre source de redondance grâce à la consistance temporelle. Il est normal de s'attendre à retrouver des patches similaires le long des trajectoires temporelles même dans des images vidéo distantes dans le temps. Il semble intuitif que les méthodes à patches peuvent bénéficier de cette source de similarité. Malheureusement, la recherche est cependant faite de façon locale pour chaque patche à cause de contraintes de temps de calcul évidentes.

Dans ce chapitre nous nous concentrons sur la recherche de patches. Nous présentons une recherche qui est une bonne approximation de la recherche exhaustive globale et nous montrons son impact sur le débruitage vidéo. Cela permet pour la première fois d'estimer l'impacte de la recherche globale sur les performances de débruitage vidéo.

La Figure 14 montre que la méthode proposée a un comportement similaire à une recherche exhaustive globale pour une fraction du coût de calcul.

Une méthode de débruitage vidéo récursive: NL-Kalman

Contrairement aux méthodes présentées dans les Chapitres 1 and 2, nous proposons ici une méthode récursive et causale dans ce chapitre. Cela signifie que la méthode n'utilise que l'image vidéo actuelle et ainsi que la version débruitée de l'image vidéo précédente. Le but de ces méthodes est de réduire l'écart entre les méthodes de type filtrage temporel qui sont légères et les méthodes non-causales basées sur l'auto-similarité qui, bien que très efficaces, sont beaucoup plus coûteuses.

La méthode proposée, appelée NL-Kalman, considère que la vidéo est un ensemble de trajectoire de patches qui peuvent se superposer. Selon une approche bayesienne, chaque trajectoire est modélisée comme un modèle linéaire dynamique gaussien et débruitée par un filtre de kalman. Pour estimer les différents paramètres, les patches similaires sont regroupés et on considère que leurs trajectoires partagent les mêmes paramètres. Le filtrage est principalement temporel. L'information spatiale similaire n'est utilisée que pour estimer les paramètres. Cette méthode permet d'obtenir des résultats comparables (en PSNR et SSIM) aux méthode de l'état de l'art utilisant plusieurs images vidéo pour chaque image débruitée. La méthode proposée a cependant une meilleure consistance temporelle. Bien que la consistance temporelle ne peut pas être montrée sans la vidéo, la Figure 15 montre que cette méthode préserve plus de détails que les autres.

Débruitage vidéo rapide sur carte graphique

Nous montrons dans les chapitres précédents que le débruitage est une étape essentielle de n'importe quelle chaîne de traitement de l'image ou de la vidéo. Malheureusement, en général à cause de contraintes de calculs, les méthodes modernes de débruitage sont rarement considérées pour être intégrée dans les chaînes de traitements. Ces algorithmes n'ont souvent que des implémentations processeur ou des implémentations carte graphique sous-optimales. Dans ce chapitre nous proposons une nouvelle implémentation carte graphique efficace de NL-means and BM3D et pour la première fois, à notre connaissance, l'algorithme de débruitage vidéo VBM3D. L'efficacité de ces implémentations permet de les utiliser dans des scénarios temps-réel. La Table 1 montre l'impact d'une implémentation optimisée pour le débruitage vidéo.

Débruitage vidéo avec un réseau de neurones convolutionnel basé sur l'auto-similarité

Les méthodes à patches étaient jusqu'à récemment l'état de l'art pour le débruitage d'image mais ont depuis été dépassées par l'apprentissage et en particulier les réseaux convolutionnels. Ces méthodes sont cependant encore l'état de l'art pour le débruitage vidéo car la redondance de l'information dans la vidéo est très important pour atteindre de bonnes performances de débruitages. Le problème est que les architectures sont en général incompatibles avec l'ajout d'information auto-similaire.

Dans ce chapitre nous proposons une nouvelle manière d'intégrer de l'information autosimilaire dans un réseau de neurone convolutionnel. L'information non-locale est introduite dans le réseau grâce à une première couche, qui n'est pas apprise, qui cherche les patches les plus proches dans une région d'intérêt pour chaque patche de l'image de référence. Les pixels centraux de ces patches sont ensuite fusionnés en un vecteur associé à chaque pixel de l'image. Cette information est ensuite donnée à un réseau convolutionnel qui est entraîné pour prédire l'image débruité. Nous avons appliqué l'approche proposée aux images et aux vidéos. Pour les vidéos, les patches sont cherchés dans un volume spatio-temporel. L'architecture proposée, montrée dans la Figure 16, atteint les performances de l'état de l'art.

Débruitage vidéo sans connaître le modèle du bruit grâce un apprentissage image vidéo à image vidéo

Dans les chapitres précédents, le modèle de bruit a toujours été considéré connu et supposé qu'il n'évoluait pas dans le temps. Cependant, modéliser la chaîne de traitement qui a produit une vidéo est une tâche de rétro-ingénierie difficile, même lorsque la caméra est connue. Cela rend le traitement de la vidéo une tâche encore plus difficile. Nous proposons dans ce chapitre une méthode de débruitage qui ne dépend pas du modèle de bruit, avec une version en ligne et une version hors ligne. Cela est possible par le raffinage d'un réseau pré-entraîné pour du bruit gaussien blanc sur une vidéo grâce une stratégie d'apprentissage image vidéo à image vidéo. Notre débruiteur peut être utilisé sans avoir aucune connaissance sur la provenance de la vidéo et des traitements appliqués après l'acquisition. Le processus en ligne n'a besoin que de quelques images vidéo avant d'arriver à produire des résultats agréables visuellement pour un large éventail de dégradations. Cette méthode atteint même les performances de l'état de l'art pour du bruit gaussien, et peut être utilisée hors ligne pour encore de meilleurs résultats. La flexibilité de la méthode proposée est illustrée dans la Figure 17.

Démosaïquage par méthode à apprentissage profond

La plupart des caméras ne capturent qu'une seule couleur par pixel. L'image ainsi produite est alors mosaïquée et doit être interpolée pour avoir les trois couleurs pour chaque pixel. Cette étape de passage d'une image mosaïquée à une image couleur est appelée démosaïquage. Ce chapitre étudie deux récentes méthodes de démosaïquage basées sur des réseaux de neurones convolutionnels qui produisent des résultats sans aucun artefacts: Deep joint demosaicking and denoising de Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] et Color image demosaicking via deep residual learning de Tan et al. [START_REF] Tan | Color image demosaicking via deep residual learning[END_REF]. Ces méthodes sont meilleures de presque deux décibels que les méthodes qui ne sont pas basées sur les réseaux de neurones, tout en étant plus rapide d'un ordre de magnitude. Un exemple très difficile de démosaïquage est montré dans la Figure 18. Pour cet exemple, seules les méthodes basées sur les réseaux de neurones produisent de bons résultats. Cela sonne le glas pour les méthodes qui ne sont pas basées sur l'apprentissage pour le démosaïquage. 12. Détection de falsifications de type copié-collé basée sur des descripteurs parcimonieux

La méthode présentée dans le Chapitre 11 surdétecte les régions similaires sans qu'elles soient falsifiées. Cela est due à la difficulté du problème et en particulier parce que les images peuvent contenir des objets similaires sans forcément avoir de falsifications. Ces objets similaires sont détectés parce que la plupart des méthodes ne sont pas assez robustes pour pouvoir discerner entre des copies et des objets similaires. La question qui se pose est comment ne pas détecter

Publications

Ce travail a mené aux publications suivantes: Patch-based video denoising

VBM3D and its extensions

VBM3D is a popular video denoising method. It is an extension to video of the well-known image denoising algorithm BM3D, which takes advantage of the sparse representation of stacks of similar patches in a transform domain. The extension is rather straightforward: the similar 2D patches are taken from a spatio-temporal neighborhood which includes neighboring frames. In spite of its simplicity, the algorithm offers a good trade-off between denoising performance and computational complexity. In this chapter we revisit this method, providing an open-source C++ implementation reproducing the results. A detailed description is given and the choice of parameters is thoroughly discussed. Furthermore, we study and compare several extensions of the original algorithm: (1) a multi-scale implementation, (2) the use of 3D patches, (3) the use of optical flow to guide the patch search. These extensions allow to obtain results which are competitive with even the most recent state of the art.

Introduction

VBM3D was proposed by [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF] as an adaptation to video denoising of BM3D, the successful image denoising algorithm [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF]. The algorithm is designed for additive white Gaussian noise (AWGN) with zero mean and standard deviation σ, i.e.

v(x) = u(x) + n(x), n(x) ∼ N (0, σ 2 ),
where x is a spatio-temporal position in the video domain (a pixel), v is the noisy video and u the unknown clean video. We consider that the v consists of f + 1 frames written v i for i between 0 and f . A patch is a small rectangular piece of the video, for example of size 8 × 8 × 3 (8 pixels width and height, 3 pixels in the temporal dimension). Patches are represented by bold lowercase letters, e.g. p. If we need to emphasize the location of the patch we write p(x), where x represents the location on the video of the top-left-front pixel of the patch. The denoising principle of VBM3D (and BM3D) is based on the redundancy of similar patches. Groups of similar 2D patches are assembled in a 3D stack of patches. A separable 3D transform is applied to this stack. The stack is denoised by applying a shrinkage operator to the coefficients in the transformed domain. The algorithm follows four basic steps:

1. Search for similar patches in the sequence, grouping them in a 3D stacks, 2. Apply a 3D linear domain transform to the 3D block, 3. Shrink the transformed coefficients, 4. Apply the inverse transform, 5. Aggregate the resulting patches in the video.

The underlying idea here is that, due to the high redundancy of a stack of similar patches, the energy will be concentrated in a few coefficients of the transform, while the noise is spread evenly among all coefficients. This allows to jointly denoise the patches of each stack. The reconstruction of the estimated video is obtained by aggregating for each pixel the estimated patches that contain it. This principle is applied twice. The first time the patches are denoised using a hard threshold in the transformed domain. In the second iteration a Wiener filter is used, with Wiener coefficients computed using the output of the first step as oracle. For the parameters of the method we will use the same notation as in [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF]. The different parameters for the patch search are the number N of patches to return, the number of temporal frames N f that are being searched, the search region for the reference frame N s , the search region for the other frames N pr , the maximum number N b of patches to compute for each local search, a correcting factor d and a maximum distance threshold τ . The threshold parameter in the first step is λ 3D . We also write ⊗ for the element-wise product. Similar parameters will be used for the hard thresholding first step and for the Wiener filtering second step. Because they can have different values for the different steps, they'll be differentiated using the subscript hard and wien respectively.

There exist other adaptations to video and 3D images of this framework: BM4D [MKEF13] and VBM4D [START_REF] Maggioni | Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms[END_REF]. These methods stack similar 3D spatio-temporal patches in a 4D stack. The main difference between them is that VBM4D uses motion-compensated patches whereas BM4D aims at denoising volumetric images (such as those appearing in medical imaging), where the 3rd dimension is just another spatial dimension. In [START_REF] Arias | A comparison of patch-based models in video denoising[END_REF] however, it was shown that even non-motion-compensated 3D patches provide a very good denoising performance.

In this chapter, we revisit the VBM3D method, providing an open source implementation and we discuss some variants introduced in [AFM18a], such as 3D patches (as in BM4D [START_REF] Maggioni | A nonlocal transform-domain filter for volumetric data denoising and reconstruction[END_REF]), optical flow guided patch search, and a multiscale implementation. In the next section, the algorithm itself is reviewed. In the following sections we consider three possible extensions: multiscale, spatio-temporal patches and motion-compensated patch search using an optical flow. Finally the performance of the algorithm is compared against other state of the art algorithms.

1.2 VBM3D : the algorithm VBM3D performs two steps, as its image counterpart BM3D. The first hard-thresholding step computes a a basic estimate which is refined in the Wiener filtering step, yielding the final estimate. Figure 1.1 represents the global structure of the algorithm. The pseudocode of the core of the algorithm is presented in Algorithm 5.

The patch search

The groups of similar patches are built by selecting a reference patch and searching around it for its nearest neighbors. This patch search is the main difference between BM3D and VBM3D. The image version of the algorithm searches a square 2D window centered at the reference patch. While the same could be done in a space-time volume for videos, Dabov et al. [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF] propose a predictive search heuristic to reduce the size of the search window. The idea behind the proposed search is to track patches in the video. Let p and q be two patches at positions (x, y, t) and (x , y , t ) respectively. The distance between p and q used to find the nearest neighbors for a given regularizing factor δ is

d(p, q) = p -q 2 2 -δ if x = x and y = y p -q 2 2 otherwise . (1.1)
This distance regularizes the patch trajectories by assuming that the patch does not move. Suppose that the reference patch is p located at (x, y, t). The goal of the proposed search is to find (at most) N patches in a window of 2N f + 1 frames around frame t. The steps are the following:

1. Find the N b nearest neighbors to p in frame t in a square search region of size N s × N s centered at the location of the reference patch (x, y, t). Let L t be the set of found patches.

2. Find the N b nearest neighbors to p in frames t = t + 1, ..., t + N f . The search region at t is the union of N pr × N pr square regions centered at positions of the candidates in L t -1 , where N pr < N s . This is depicted in Figure 1.2.

3. Similarly, find N b nearest neighbors to p in frames t = t -1, t -2, ..., t -N f . This time the search region is the union of squares centered at the positions of the patches in L t +1 .

4. Remove the candidates with distance (1.1) larger than τ from

L = t+N f t =t-N f L t ,
the set combining all the candidates computed in each frame.

5. Keep the best N candidates from L.

6. Because the next steps of the algorithm (in particular the domain transforms) require a number of patches that is a power of 2, only the largest power of 2 smaller or equal to N is kept.

These steps are summarized in Algorithms 1 and 2.

Patch stack filtering: hard-threshold step

The first step processes each frame by filtering stacks of similar patches and aggregating them in an output image. The reference patches for the stacks are all patches on a sub-grid of step st hard .

For each group there is first an estimation, followed by an aggregation.

Algorithm 1: compute_similar_patches: VBM3D patch search input : Noisy video v = (v 0 , . . . , v f ), a reference patch p, the number of patches to return N , a number of temporal frames N f , a search region for the reference frame N s , a search region for the other frames N pr , the maximum number of patches to compute with each local search N b , the size of the patch k, a correcting factor d, a maximum distance threshold τ output : L the list of the N patches closest to p and their distance 1 t ← frame at which p is located 2 L t ← local_search(p, p, N s , k, N b , d, u t ) // Centered on p using p as a reference // Search in the N f following frames

3 for t f = (t + 1) to min(t + N f , f -1) do 4 for q ∈ L t f -1 do 5 L t f ← local_search(q, p, N pr , k, N b , d, u t f ) // Centered on q
using p as a reference // Search in the N f previous frames

6 for t p = t -1 to max(t -N f , 0) do 7 for q ∈ L tp+1 do 8 L tp ← local_search(q, p, N pr , k, N b , d, u tp ) // Centered on q using p as a reference 9 L ← i∈ t-N f ;t+N f L i
L ← elements from L with distance smaller than τ // the 3D transform requires a number of patches which is a power of 2 N l ←closest power of 2 small or equal than min(sizeof L, N ) if L has more than N l elements then L ← N l best candidate from L return L Algorithm 2: local_search: Local search input : A patch p center of the search region, a reference patch p , the search region size s, the size of the patch k, the number of patches to return N b , a correcting factor d, a frame u output : L the list of the patches closest to p in the region described by p and s and their distance // Compute the distances for all the patches in the local region 1 for each patch q of size k × k in the spatial region of size s × s centered on p do 2 if q is at the same spatial position than p then Estimation. For a given reference patch p, we first find its similar patches L as described in Section 1.2.1. These patches are stacked in a 3D volume P(p), of size k × k × N . The first spatial slice of this stack is the reference patch, and the remaining ones are the similar patches in L ordered by their distance to p. A 3D domain transform is first applied to the group of patches followed by a thresholding of the resulting spectrum (excepting the DC component of every patch). The inverse 3D domain transform is then applied to the thresholded coefficients to obtain the estimation, i.e.

3 Add ( q -p 2 2 -d, q) to L 4 else 5 Add ( q -p 2 2 , q) to L 6 Sort L
P(p) = T -1 3D (HT λ (T 3D (P(p)))) (1.2)
where HT is defined by

HT λσ (ν) = ν if ν is a DC component or |ν| > λσ 0 otherwise. (1.3)
In practice the 3D transforms are chosen to be separable, consisting of a 2D spatial transform applied directly on the patches of P(p), typically a bi-orthogonal wavelet transform, followed by an 1D transform along the third dimension of the stack, typically a Haar transform. The pseudocode of the estimation for the first step is presented in Algorithm 3.

Aggregation. An output pixel is estimated several times, since it belongs to several patches and each patch can be estimated multiple times (once for each group it belongs to). To compute the output frame u these estimates are aggregated. For a pixel ρ of the frame, this aggregation is performed as

u(ρ) = patch p of v q∈ P(p) ω p (ρ)q(ρ) patch p of v ω p (ρ) (1.4)
where a patch p has a support of the size of the image and is zero everywhere except on the actual location of the patch. The weights ω used during the aggregation have two contributions. A "perstack" contribution w ht computed during the estimation is

w ht (p) = 1 σ 2 N hard (p)
where N hard (p) corresponds to the number of coefficients that have not been thresholded during the estimation of P(p). Each coefficient w ht (p) is properly defined because N hard (p) is always positive as the DC component is never thresholded. These weights penalize estimates with large variance, as proposed in [START_REF] Guleryuz | Weighted overcomplete denoising[END_REF]. In addition a "per-pixel" aggregation weight is used to smooth out the blocking artifacts that result from working with patches. This weight comes from a 2D Kaiser window K(p) of size k hard × k hard (see Eq. (1.6)) located on the position of p. The Kaiser window of size L x × L y of parameter β is

K(p)(x, y) = I 0 β 1 -(2x/L x ) 2 I 0 β 1 -(2y/L y ) 2 /I 0 (β) 2
(1.5) with 0 x L x , 0 y L y and I 0 the zeroth-order modified Bessel function (1.6)

Finally the (patch) weight is defined by ω p = w ht (p)K(p). The pseudocode with the aggregation step can be found in Algorithm 5.

Algorithm 3: ht_filtering: Hard thresholding input : A group of similar patches L, a 3D transform T 3D , the noise variance σ 2 output : A list of filtered patches L, the aggregation weight

ω 1 L ← T 3D (L) 2 n ← 0 3 for each patch p in L do 4 for each pixel ρ of p do 5 if p(ρ) > λσ or ρ is the DC component then 6 n ← n + 1 7 p(ρ) ← p(ρ) 8 else 9 p(ρ) ← 0 10 ω = 1 σ 2 n 11 L ← T -1 3D (L) 12 return L, ω
Algorithm 4: wiener_filtering: Wiener thresholding input : A group of similar patches L, a first estimate of L called L , a 3D transform T 3D , the noise variance σ 2 output : A list of filtered patches L, the aggregation weight ω

1 L ← T 3D (L) 2 ω ← 0 3 for each patch p in L, p the corresponding patch in L do 4 for each pixel ρ of p do 5 α ← p (ρ) 2 p (ρ) 2 +σ 2 6 p(ρ) ← αp(ρ) 7 ω ← ω + α 2 8 ω ← 1 σ 2 ω 9 L ← T -1 3D ( L) 10 return L, ω

Patch stack filtering: Wiener filtering step

The second step of the algorithm uses the basic estimate computed during the first step for the patch search to compute the coefficients of a Wiener shrinkage of the transformed coefficients.

The video is processed by building groups of patches around reference patches from a coarse subgrid with step st wien .

Estimation. Given a reference patch p, a group of similar patches is selected as described in Section 1.2.1, but using patches extracted from the basic estimate for computing the patch distance. Two sets of patches are extracted: one from the noisy sequence and the other one from the basic estimate at the same locations. Each set of patches is stacked in a 3D volume: P(p) for the noisy patches and P(p) for patches of the basic sequence. The noisy stack is denoised with a Wiener filter. The filtering coefficients are estimated from P(p). Just like the first step, a 3D domain transform is first applied to the group of patches before the application of the Wiener filter. The computation is done following Equation (1.7).

P(p) = T -1 3D (W F (T 3D (P(p)))) (1.7)
where W F on a frequency ν (with ν corresponding to the same frequency in the basic estimation) is defined by

W F (ν) = ν2 ν2 + σ 2 f (1.8)
In practice the 3D transforms are chosen as a 2D transform applied directly on the patches of P(p) and P(p), typically a DCT, followed by an 1D transform along the third dimension of the group, typically a Haar transform. The pseudocode of the estimation for the first step is presented in Algorithm 4.

Aggregation. Just as with the first step, the estimation gives multiple estimates per pixel. Therefore the different estimates are aggregated using the same principle as in Section 1.2.2, defined by Equation (1.4). The only difference lies in the weights. The weights ω used during the aggregation are computed in part during the estimation. The part w wf computed during the estimation

is w wf (p) = 1 σ 2 ρ pi (ρ) 2 pi (ρ) 2 +σ 2 2 -1
which is actually linked to the squared 2 norm of the vector of coefficients used for the filtering. The rest of the weight come from a 2D Kaiser window K(p) of size k wien × k wien applied to avoid boundary effects from the patches. Finally the pixel weight is defined by ω p (x) = w wf (p)K(p). The pseudocode with the aggregation step can be found in Algorithm 5.

Reproducing the original VBM3D

We now compare the results obtained with our implementation and with the binaries released by the authors of the original VBM3D [DFE07]1 . Throughout this work we will use a test set of seven grayscale test sequences obtained from the Derf's Test Media collection2 . The original sequences are of higher resolution and in RGB. They have been downscaled and converted to grayscale.

Table 1.1 compares the PSNR obtained by our implementation to that of the original binaries. Our results are slightly below: The average gap starts at 0.27dB for σ = 10 and reduces to 0.18dB for σ = 40. Station and sunflower are the sequences where the gap is larger, reaching 0.68dB for the later at σ = 10. A visual comparison of both results is shown in Figure 1.3, for noise σ = 40. Both results are visually very similar, with the same type of artifacts. A closer inspection reveals that the result of the original VBM3D is slightly smoother, resulting in less noticeable Algorithm 5: VBM3D algorithm input : A video v, the noise variance σ 2 , the number of similar patches to compute N , a number of temporal frames N f , the size of the search region in the reference frame N s , the size of the search region in the other frame N pr the number of patches kept in each frame N b , the size of the patch k, d, the distance threshold τ , the thresholding parameter λ 3D , the domain transform T 3D , the coefficient fo the Kaiser window β, the step of the grid on which the patch are taken st output : An final estimate denoised video v(2) 1 K 1 ← Kaiser window of size k hard and coefficient β hard 2 K 2 ← Kaiser window of size k wien and coefficient β wien // Step 1 3 for each p on the grid of step st hard do // Search for similar patches in the noisy video

4 L p ← compute_similar_patches(v, p, N hard , N f , N s , N pr , N b , k hard , d hard , τ hard )
// Filter the group of patches using a hard thresholding

5 ( Lp , ω) ← ht_filtering(L p , T 3D,hard , σ 2 ) 6 for q ∈ Lp do 7 a(q) ← a(q) + ωK 1 ⊗ q 8 w(q) ← w(q) + ωK 1 9 for each pixel x do v(1) (x) ← a(x)/w(x) //
Step 2 for each p on the grid of step st wien do // Search for similar patches in the basic estimate

L p ← compute_similar_patches(v (1) , p, N wien , N f , N s , N pr , N b , k wien , d wien , τ wien )
L p ← patches from v at the same position than the one from L p // Filter the group of patches using a Wiener filtering DCT artifacts and on some textures with decreased contrast (see for instance the grass in the rightmost figure). ) to guide the search region. However, it has been observed that using optical flow to shape the search region can still be beneficial for patch based methods [START_REF] Arias | Video denoising via empirical bayesian estimation of space-time patches[END_REF][START_REF] Arias | A comparison of patch-based models in video denoising[END_REF], as it allows to find better matches. For a pair of two images A and B, the optical flow aims at finding a vector field o(x, y) = (δ x , δ y ) such that any point (x, y) of the image domain solves

( Lp , ω) ← wiener_filtering(L p , L p , T 3D,wien , σ 2 ) for q ∈ Lp do a(q) ← a(q) + ωK 2 ⊗ q w(q) ← w(q) + ωK 2 for each pixel x do v(2) (x) ← a(x)/w(x) return v(2)
A(x + δ x , y + δ y ) = B(x, y).
(1.9)

The displacement vector can be sub-pixel, in which case the image A needs to be interpolated. We decided to use the TVL1 optical flow method [START_REF] Zach | A duality based approach for realtime tv-l 1 optical flow[END_REF], in particular the implementation provided in [START_REF] Sánchez Pérez | TV-L1 Optical Flow Estimation[END_REF]. We compute the optical flow in a downscaled version of the video by a factor of 4, and scale it back to the original resolution. This reduces the running time and the impact of the noise while still having a reasonable precision, as shown in [START_REF] Ehret | Non-local kalman: A recursive video denoising algorithm[END_REF].

We add the optical flow to VBM3D as a guide, the same way it is done for VNLB [START_REF] Arias | Video denoising via empirical bayesian estimation of space-time patches[END_REF]. The spatio-temporal search region is defined as two sequences of N f square windows of size N pr × N pr (plus the N s × N s window corresponding to the reference frame), whose centers follow the motion trajectory of the reference patch. The trajectory is estimated using the forward and backward optical flow. We use the center corresponding to the position of the reference patch propagated in previous, respectively following, frames using the backward, respectively forward, optical flow. The forward half of the trajectory ϕ x,y,t passing through (x, y, t) is computed by integrating the forward optical flow o f (also indexed by time on top of the spatial position) as follows:

ϕ x,y,t (h) = o f ([ϕ x,y,t (h -1)], h -1) + ϕ x,y,t (h -1), h = t + 1, . . . , t + N f , (1.10)
where [ • ] and • denote the round and floor operators. The backward half of the trajectory is defined analogously using the backward optical flow o b . This also means that only one center needs to be tracked, in contrast to the N b required by a regular VBM3D search. Parameters are kept the same as in the original algorithm. The guided version of the search is summarized in Algorithm 6.

We also tested setting the parameter d to zero for both steps since one can assume that it would be redundant with the regularization of the patch search offered by the optical flow. PSNR results are shown in Table 1.2. Using the optical flow as a guide greatly improves the quality of the denoising. The non-zero d yields better results both with and without the optical flow guide, thus we decided to keep it in our final version.

Figure 1.4 shows results obtained with the different extensions that will be described in this section. The first column corresponds to VBM3D with the default parameters, and the 3rd column to the one using an optical flow guided search region (denoted "VBM3D OF"). Guiding the patch search allows to recover more details. This is because the tracking of patches is more robust to motion than the block-matching suggested for the original VBM3D and therefore provides better matches. Moreover, the optical flow is not required to be very precise since it is used only as a guide for the center of the search region.

Algorithm 6: compute_similar_patches: VBM3D patch search guided by the optical flow input : Noisy video v = (v 0 , . . . , v f ), a reference patch p, the number of patches to return N , a number of temporal frames N f , a search region for the reference frame N s , a search region for the other frames N pr , the maximum number of patches to compute with each local search N b , the size of the patch k, a correcting factor d, a maximum distance threshold τ , the forward and backward optical flows o f and o b output : L the list of the N patches closest to p and their distance 1 t ← frame at which p is located

2 L t ← local_search(p, p, N s , k, N b , d, u t )
// Search in the N f following frames 3 q ← p 4 for t f = (t + 1) to min(t + N f , f -1) do 5 q ← o f (q, t f -1) Follow the center using the forward optical flow

6 L t f ← local_search(q, p, N pr , k, N b , d, u t f ) // Search in the N f previous frames 7 q ← p 8 for t p = t -1 to max(t -N f , 0) do 9 q ← o b (q, t p + 1
) Follow the center using the forward optical flow

L tp ← local_search(q, p, N pr , k, N b , d, u tp ) L ← i∈ t-N f ;t+N f L i L ←

Spatio-temporal patches

The patch similarity is determined based on the (squared) Euclidean distance between patches. Due to the noise, the Euclidean distance follows a non-central χ 2 distribution, with variance

var 1 m q 1 -q 2 2 = 8σ 2 m σ 2 + 1 m p 1 -p 2 2 ,
where q 1 , q 2 are the noisy versions of the patches p 1 , p 2 , and m denotes the number of pixels in the patch. The noise in the distance can be reduced by considering larger patches. However, increasing the spatial size of the patch also increases the distances between patches and reduces the likelihood of finding similar ones. Adding an additional temporal dimension in a spatiotemporal patch allows to increase the number of pixels in the patch, without increasing its spatial size. Due to the high redundancy of the video in the temporal dimension, increasing the temporal size of the patch causes a much lower increase in the patch distances. In practice, a spatiotemporal patch can be seen as a set of 2d patches from consecutive frames. Excepted in the case of motion compensated patched presented right after, these 2d patches are centered on the same spatial position in each frame. When the motion is known or can be estimated, then it is natural to consider motioncompensated spatio-temporal patches (see for instance [START_REF] Maggioni | Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms[END_REF][START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF]). Alternatively, rectangular spatio-temporal patches with no motion compensation have been also used [START_REF] Protter | Image sequence denoising via sparse and redundant representations[END_REF][START_REF] Arias | Video denoising via empirical bayesian estimation of space-time patches[END_REF]. For more complex types of motion, using rectangular spatio-temporal patches will result in a larger variability in the set of nearest neighbors of a given patch, due to the fact that both the spatial texture and the motion pattern may vary. At least in principle, better results should be obtained using motion compensation. However, in practice, for higher levels of noise the bad quality of the estimated motion can undermine the final result.

The principle of BM3D has been applied to 3D patches with and without motion compensation. VBM4D, introduced in [MBFE12], uses motion-compensated spatio-temporal patches for video denoising (the "V" stands for video). The motion is estimated using block matching. BM4D uses cubic patches without motion compensation (of size 4 × 4 × 4 or 5 × 5 × 5), aiming at filtering volumetric images [START_REF] Maggioni | A nonlocal transform-domain filter for volumetric data denoising and reconstruction[END_REF]. In [START_REF] Maggioni | A nonlocal transform-domain filter for volumetric data denoising and reconstruction[END_REF] the authors compare the performance of both VBM4D and BM4D on video denoising concluding that VBM4D was the best video filtering strategy.

However, in [START_REF] Arias | Video denoising via empirical bayesian estimation of space-time patches[END_REF][START_REF] Arias | A comparison of patch-based models in video denoising[END_REF] it is shown that rectangular spatio-temporal patches with a temporal size of only two frames improve the denoising quality and still provide higher temporal consistency than a 2D patch. Based on those results, in this chapter we evaluate rectangular spatio-temporal patches of size 8 × 8 × 2 in the first step and 7 × 7 × 2 in the second (i.e we keep the spatial patch size).

In Table 1.3 we compare the quantitative results obtained by using spatial and spatio-temporal patches (denoted by "ST" in the table). We also consider the effect of guiding the patch search using an optical flow, indicated as "OF".

The first four columns of Figure 1.4 show results with/out motion-compensated search and spatio-temporal patches. From a qualitative point of view, using spatio-temporal patches provides better temporal consistency. In addition, the patch distance is more reliable since the number of pixels in the patches is doubled. This helps retrieve details and textures for regions with a simple motion (e.g. translational). For low noise levels, the effect of these 3d patches is mixed. While the results seem more consistent temporally, they are blurrier for sequences with complex motions. This explains the drop in PSNR observed for pedestrians, sunflower and touchdown between VBM3D and VBM3D ST for σ = 10. As the noise level increases this detail loss is out-weighted by the increased robustness to noise of the patch matching.

When spatio-temporal patches are used in conjunction with the optical-flow-guided search, their positive impact is magnified. Although the patches are not themselves motion-compensated, having a motion-compensated search region helps find better matches, even in sequences with more complex motion patterns. The motion-compensated search region also improves the temporal consistency, although to a lesser extent than the spatio-temporal patches. The best result, both in terms of PSNR and temporal consistency, is obtained when both strategies are used together (VBM3D ST+OF).

Multiscale video denoising

Multiscale approaches have shown to both reduce the residual noise but also improve the visual quality of the result. Indeed, most denoising algorithms work by processing local regions of the image/video (a patch, a group of patches, the receptive field of a neuron, etc). As a result, these methods fail to remove the lower frequencies of the noise. This results in smooth bumps mostly noticeable in large non-textured areas. Multiscale approaches are able to reduce this artifact by applying the denoising algorithm at different scales.

There are two main approaches for multi-scale denoising in the literature. The first one consists in modifying the denoising algorithm to consider several scales of the image/video. See for instance the multiscale version of the EPLL method [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF] proposed by Papyan and Elad [START_REF] Papyan | Multi-scale patch-based image restoration[END_REF]. Another approach proposed in [START_REF] Facciolo | Conservative scale recomposition for multiscale denoising (the devil is in the high frequency detail)[END_REF][START_REF] Pierazzo | Multi-Scale DCT Denoising[END_REF] considers the denoising algorithm as a black box, applying it as is at multiple scales. The result is then obtained by merging the results of each scale. This has the benefit that it can be applied to any denoising method, without any adaptation needed.

The multiscaler of [START_REF] Facciolo | Conservative scale recomposition for multiscale denoising (the devil is in the high frequency detail)[END_REF] first creates a pyramid from the noisy input image v 0 = v by applying a downscaling operator D v s+1 = D(v s ), for s = 0, ..., S -1.

(1.11)

At each scale, the denoising algorithm is applied and yields denoised images ûs . These are then recomposed into a single multiscale output ûms . The recomposition is recursive. The recursion is initialized at the coarsest level by defining ûS ms = ûS , and then proceeds as follows:

ûs ms = ûs -U(L(D(û s ), f rec )) + U(L(û s+1 ms , f rec )), (1.12)
where U is the upscaling operator, and L( • , f rec ) is a low-pass filter with cutoff frequency parameterized by f rec (f rec does not depend on the scale). This equation substitutes the low frequencies of the single-scale result ûs with the multiscale solution ûs+1 ms , computed using the coarser scales s + 1, ..., S. The recomposition parameter f rec determines which low frequencies are substituted. In one extreme, the low-pass filter lets all frequencies pass, in which case the whole coarse solution is used. At the other end, f rec filters out all frequencies: the solution at the coarser level ûs+1 ms is discarded, and the output of the recomposition is the single scale denoising û0 . In [START_REF] Facciolo | Conservative scale recomposition for multiscale denoising (the devil is in the high frequency detail)[END_REF] it is found that the optimum is to filter out some of the high frequencies of the coarser level ûs+1 ms . However, the exact amount depends on the denoiser.

To apply the multiscaler on a video, we apply it spatially, i.e. the temporal dimension is not downscaled. We first create a spatial pyramid of the entire video by creating a pyramid of each frame. We then denoise these videos, and recompose them by applying Eq. (1.12) to each frame. This is summarized in Algorithm 7. The parameters of the multiscaler are the number of scales, the downsampling ratio and the recomposition parameter f rec . We shall set the downscaling ratio to 2 (the default), and try different values for the number of scales and the recomposition factor. There are different possibilities for the down/upscaling operators and the low-pass filter.

DCT pyramid. The DCT multiscaler uses a DCT pyramid which guarantees white Gaussian noise at all scales. The downscaling is performed by computing the DCT transform of the image and keeping only the quadrant of the image corresponding to the lowest frequencies. The upscaling is done by zero-padding in the DCT domain, and the low-pass filtering zeroes out the highest frequencies in the DCT domain. In this case f rec represents the ratio of frequencies that are left: f rec = 1 corresponds to an all-pass filter, where as f rec = 0 filters out all the image.

Laplacian pyramid. The downscaling and upscaling operators samples the image using a Lanczos kernel:

k a (x) = sinc(x)sinc(x/a) if |x| < a 0 otherwise. (1.13)
We set a = 3. For the downscaling, to reduce aliasing, the image is downsampled using a scaled version of the kernel k 3 ( • /2) (as described in [START_REF] Schumacher | General filtered image rescaling[END_REF]). 3 The low pass filter used is the Gaussian filter of width f rec . When f rec → ∞ we obtain the single scale output, and if f rec = 0 no frequencies from the coarser solutions are discarded.

Figures 1.5 and 1.6 show plots of the average PSNR for our seven test sequences obtained with the two multiscalers varying the number of scales and the recomposition cutoff parameter f rec . In each figure, we show the results of the multiscaler applied to the standard VBM3D, and to the one with statio-temporal patches and guided patch search (VBM3D ST+OF).

Both multiscalers have a positive impact when applied to the standard VBM3D. The gain can be up to 0.3dB for noise 10, 0.5dB for noise 20 and 0.8dB for noise 40. However, when applied to the VBM3D ST+OF version of the algorithm, the multiscaler does not improve the result. In fact, for noise 10 and 20, the best PSNR is attained by the single scale algorithm and the PSNR deteriorates as the cutoff frequency of the low-pass filter is increased (i.e. as more frequency components from the coarse solution are used).

The multiscaler achieves a better denoising of large objects with smooth textures by removing low-frequency noise left by the denoiser. This low-frequency noise is much stronger for the VBM3D denoiser than for the VBM3D ST+OF version. Hence, the improvement provided by the multiscaler is smaller for the latter. This also depends on the characteristics of the sequence. Frames with smooth objects occupying larger areas will benefit from the multiscaler. Yet, the multiscaler introduces artifacts penalizing the PSNR (particularly additional ringing for the DCT multiscaler). These artifacts are not temporally coherent and can therefore be quite noticeable.

Based on these plots we chose to select the Lanczos3 multiscaler. We use 2 scales and a recomposition factor f rec = 1 when applied to VBM3D ST+OF (i.e. VBM3D ST+OF+MS) and 3 scales with recomposition factor f rec = 0.6 when applied to the standard VBM3D (VBM3D MS). Table 1.4 shows the obtained PSNRs. The visual results of VBM3D ST+OF+MS are shown in Figure 1.4. The impact of the multiscaler can be noticed in the top row (results for pedestrian) as a reduction of the low-frequency noise in the smooth areas in the image. For the other sequences, since they are highly textured, the effect of the multiscaler is subtle. A careful examination reveals some texture loss.

Comparison with the state of the art and conclusion

In Table 1.5, we compare the PSNR of different recent denoising methods with VBM3D as well as with versions of VBM3D modified using different combinations of the improvements suggested in Sections 1.3.3, 1.3.2, 1.3.1. We included the patch-based methods SPTWO [START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF] and VNLB [START_REF] Arias | Video denoising via empirical bayesian estimation of space-time patches[END_REF] and VNLnet [DEF + and VNLB methods are based on grouping similar patches and filtering them on a transformed domain. However, instead of a pre-defined transform, an optimal transform is estimated for each group (the principal directions of the group of patches). While this gives good results, it requires computing an SVD of the matrix formed by the similar patches, resulting in very slow algorithms. VNLB groups 3D patches 10 × 10 × 2, similar to our spatio-temporal patches. SPTWO instead uses long 3D spatio-temporal patches which are motion compensated, producing highly temporally consistent results. VNLnet applies a CNN to a stack of frames. The frames are all registered using block matching with a large block size. It produces good results with high temporal consistency except when the block matching fails to find good matches. Since it is implemented on GPU it runs much faster than VNLB and SPTWO, although it also has a high computational cost. VBM3D combined with spatio-temporal patches and a patch-search guided with an optical flow gives very competitive results even compared to the latest state of the art, and especially for higher noises. When combined with the other improvements, it seems that multiscaling does not increase the quality of the denoising and can even degrade it in terms of PSNR. Visual examples are shown in Figures 1.4, where we compare results obtained with the original VBM3D with our implementation, using 3D patches and an optical flow to guide the search region.

In conclusion the proposed implementation of VBM3D and its variants, provide a video denoising framework represent an interesting trade-off between denoising quality and computational complexity. Most contributions in the area have focused on how to model and filter the groups of similar patches, but little attention has been given to how these groups are built. Typically similar patches are grouped by selecting a reference patch and searching exhaustively for the similar ones in a local 2D, or 3D for videos, neighborhood. The size of the search region is a parameter of the algorithm which trades off quality of the result for computational cost.

Introduction

In the case of a single image, a local search region is justified by the fact that similar patches are likely to be close to each other in the image domain. Videos however, have an additional strong source of redundancy given by the temporal consistency. A patch is expected to have similar patches along its motion trajectory, even in distant frames. It seems intuitive that patchbased methods should benefit from this larger set of similar exemplars. Some methods estimate the motion in the video to tackle this problem. A motion compensated search window can track the patch trajectories for a certain number of frames [START_REF] Liu | A high-quality video denoising algorithm based on reliable motion estimation[END_REF][START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF]. Nevertheless, the size of these search regions is still limited by the computational cost and the accumulation of errors in the estimated motion.

In this chapter we focus on the patch search. We present an efficient global approximate search technique and demonstrate its impact on video denoising. To that end we take two patchbased image denoising methods, namely BM3D [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF] and NL-Bayes [START_REF] Lebrun | A Nonlocal Bayesian Image Denoising Algorithm[END_REF] and adapt them to video (simply by searching similar patches in multiple frames instead of just the current one). We provide an extensive experimental evaluation in grayscale and color sequences. Our results show that substantial gains in performance are obtained by searching globally in the video sequence, indicating that video denoising still has significant room for improvement by using clever global search methods. In particular, the NL-Bayes method with global search outperforms state-of-the-art methods such as V-BM3D [DFE07] and V-BM4D [MBFE12] by a significant margin, and the recently proposed SPTWO [START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF] by a lower margin.

In recent years several efficient techniques for approximate nearest neighbor search have been proposed, after the introduction of the PatchMatch algorithm by Barnes et al. [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF]. These methods compute a nearest neighbor field for patches located in a dense or semi-dense grid, and use heuristics that benefit from the overlap of adjacent patches in the grid. Most of these works focus on finding the nearest neighbor, but they can be extended to handle k nearest neighbors [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF]. In practice k is kept small since even with these efficient techniques, computing a large number k of nearest neighbors for a dense grid of patches remains too costly.

In this chapter we focus on a significantly different problem: compute a large number, namely k, of nearest neighbors for a single query patch. By allowing independent queries, our technique is more flexible, and is straightforward to apply to patch-based denoising methods. In particular, this is useful for certain denoising algorithms which save computations by processing a sparse set of reference patches dynamically determined during the execution of the algorithm, and also makes parallelization easier.

The rest of the chapter is organized as follows: in §2.2 we describe briefly two state-of-the-art image denoising algorithms (NL-Bayes [START_REF] Lebrun | A Nonlocal Bayesian Image Denoising Algorithm[END_REF] and BM3D [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF]) which we selected to demonstrate our global search. In §2.3 we propose a new heuristic to accelerate approximate nearest neighbor search for patches in images and videos. A comparison with state-of-the-art methods is performed in §2.4. Concluding remarks are given in §2.5.

NL-Bayes and BM3D

BM3D and NL-Bayes are patch based methods which follow the same overall framework to denoise an image. For each patch in a set of patches to be denoised, they first search for similar patches inside the image. This search region is usually rectangular and centered on the query patch. The patches found during the search are then processed to compute an estimate of the corresponding clean patches. The denoised patches are then aggregated to create a first (also called basic) estimate of the clean image. This process is then iterated once. In the second step, the basic estimate is used as a pilot for the patch search and the processing of the similar patches.

To test the proposed global search, we consider extensions to video of these algorithms by searching for similar 2D patches in a spatio-temporal volume in the video, as proposed in [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF], [START_REF] Arias | Towards a bayesian video denoising method[END_REF]. This framework is presented in Algorithm 8.

The main difference between BM3D and NL-Bayes algorithms lies in the processing of the set of similar patches. NL-Bayes learns a Gaussian a priori model for the set of patches and computes the patches as the maximum a posteriori estimate. BM3D stacks the patches in a 3D signal which is denoised using shrinkage on a transformed domain. A more detailed presentation of BM3D is available in Chapter 4.

We use a slight modification of video NL-Bayes [START_REF] Arias | Towards a bayesian video denoising method[END_REF] which caps the rank of the patch covariance matrices for the groups of similar patches. This improves both performance and speed of this algorithm, and adds a rank parameter r to each step.

We modified the search (corresponding to steps 2 and 6 in Algorithm 8) by considering three different approaches: a local approach which uses the local search in a spatio-temporal volume centered at the reference patch for both denoising iterations; a global approach which searches in the full video volume for both steps of the algorithm; and mixed approach which uses the local search in the first step and the global in the second iteration (step 6 in the pseudocode). The global patch search heuristic is presented in Section 2.3.

Algorithm 8: Image/video denoising framework input : Noisy image/video v, noise level σ output : Estimate of noiseless image/video v 1 for all patch q in v do 2 Retrieve n nearest neighbors to q 3 Process the set of similar patches and compute a denoised estimate q of q 4 Aggregate estimated patches on v to compute the basic estimate 5 for all patch q in v do 6 Retrieve n nearest neighbors to q 7 Process the set of similar patches and compute a denoised estimate q of q 8 Aggregate estimated patches on v to compute the final estimate 9 return v

In the Section 2.4 we shall compare these approaches among them and also with other video extensions of BM3D and NL-Bayes which use sophisticated local patch search regions, V-BM3D (presented in Chapter 1) and SPTWO. V-BM4D [START_REF] Maggioni | Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms[END_REF] is an extension of V-BM3D. SPTWO [START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF] is based on NL-Bayes but performs a more elaborate search. First, the optical flow towards the adjacent six frames is computed and used to warp them to the reference frame. A s × s × 13 spatio-temporal patch is then associated to each s × s patch in the reference frame by extending its temporal dimension on the volume defined by the warped frames. Then, k patches closest to the reference are searched for in a local neighborhood. The final set of matches is given by the 13k 2D slices from the newly found extended patches (some of them might be discarded by an occlusion detection step). The usage of these extended patches reduces the noise in the distance while still keeping a small spatial patch (the patch spatial size is s = 5). Furthermore, the patches are warped by the optical flow. This is useful in cases where the motion is not translational. On the downside, this method relies heavily on the optical flow.

Heuristics for global patch search

There are already a number of efficient patch search techniques which exploit the so-called image coherency: Neighboring query patches, since they overlap, have high chances of having neighboring matches in the database image. Thus knowing the position of a good match for a patch helps in determining good matches for its neighbors. This idea was first applied by [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF] to compute a nearest neighbor field (NNF), assigning the closest k patches to each patch in the image. The original algorithm was presented for images but can easily be extended to video. More recent works reported significant improvements (between one and two orders of magnitude) by combining PatchMatch with more classic search data structures such as partition trees (more specifically KD-trees) [START_REF] Olonetsky | TreeCANN -kd-tree Coherence Approximate Nearest Neighbor algorithm[END_REF][START_REF] He | Computing Nearest-Neighbor Fields via Propagation-Assisted KD-trees[END_REF] and locality sensitive hashing [KA11, BZL + 15].

All these methods compute a dense k-NNF, typically for a small k. The reason is that when k is large (e.g. k > 20) computing a dense NNF is too costly. In such cases it is preferable (if the application allows to) to compute the k nearest neighbors for a small set of patches. In particular, for image/video denoising, a common speed-up strategy of patch-based methods is to reduce the number of query patches. For instance, in [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF] the query patches form a regular subgrid, and in [START_REF] Lebrun | A Nonlocal Bayesian Image Denoising Algorithm[END_REF] the query patches are irregularly located and are determined during the evolution of the algorithm. This is why we need a method that can compute nearest patches for a set of selected patches without having to do it for all patches of the image/video. There is a vast literature on data structures for nearest neighbor search on generic metric spaces or vector spaces; but these classical tools do not exploit the image coherency. In this section we briefly review one of such approaches, namely partition trees, and show a simple yet effective modification for a fast approximate k-nearest neighbor search of image patches.

Partition Trees

A partition tree is an inductive data structure encoding the position of a set of n points in R d (the database). Once the partition tree has been built it is used to search for the nearest neighbors of a point (the query). Nodes in the tree can either be leaves (also called bins) containing a maximum number of elements, or a split value between two subtrees: the "left" subtree and the "right" subtree. A partition tree splits recursively the data space by applying a simple split at each node (with the exception of the leaves). At each splitting operation, the set of elements in the current subtree is split in two equally sized subsets which are then used to construct the child subtrees. The construction of the tree is fully specified by a split value function and a split function. The split value function assigns a split value to a set of elements, whereas the split function assigns one of two groups to an element.

A partition tree can be directly used to search for the exact k-nearest neighbors with expected complexity of O(log(n)), where n is the number of points. In practice when the dimensionality of the elements is too large, its performance drops and becomes comparable to a linear search, this problem was shown with image patches by Kumar et al. in [KZN08]. This is indeed the case for image and video patches, so we shall rule out exact search and settle with the so-called first bin heuristic: The candidates for the k-nearest neighbors are taken only from the bin of the query patch. On its own, the first bin heuristic does not suffice to provide good quality matches, but this will be solved by combining it with other heuristics that exploit the fact that patches lie on images. This allows to be much faster than other approximative search such as FLANN [START_REF] Muja | Fast approximate nearest neighbors with automatic algorithm configuration[END_REF] for the same quality of nearest neighbors.

By randomizing the construction of these trees, forests can be built. It is then possible to perform independent parallel queries with each tree and select the best elements, therefore improving the quality of the elements retrieved using the first bin search. Since the queries can be done in parallel, this improvement does not take more time than the basic tree search.

Partition Tree Search With Local Refinement

We propose to use the first bin search in a partition tree (or a forest) to obtain a first set of m ≈ k initial candidates matches, and refine this set of candidates as follows: For each of the elements of the candidate list, we search in a small 2d local region in the image centered at the candidate. We call the resulting strategy Partition Tree with Local Refinement (PTRL). The complete pseudocode for this technique is presented in Algorithm 9. This refinement is inspired by the local refinement of PatchMatch that will be studied in Chapter 10.

Note that the proposed approximate search heuristic can in principle be applied in conjunction to other data structures for nearest neighbor search such as those based on hashing [START_REF] Andoni | Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions[END_REF][START_REF] Andoni | Beyond locality-sensitive hashing[END_REF]. We do not pursue this in the present work.

Algorithm 9: PTLR search heuristic input : v an input video, F a Partition tree forest constructed with the patches from v, p a request patch from v, κ × κ the size of local search region output : A list of matches for p 1 Retrieve the list {ϕ 1 , . . . , ϕ k } of k best matches from the forest F using the retrieval algorithm from a partition tree forest 2 for i = 1 to k do 3 Perform an exhaustive search in a local 2d region of size κ × κ centered on ϕ i for better matches 4 return the list {ϕ 1 , . . . , ϕ k } of k best matches after the update using the PTLR search

Search Parameters

The choice of the partition tree has a strong impact on the performance. The most common partition tree is the KD-tree [START_REF] Louis | Multidimensional binary search trees used for associative searching[END_REF]. However, it has been shown that VP-trees [START_REF] Peter N Yianilos | Data structures and algorithms for nearest neighbor search in general metric spaces[END_REF] produce better results when working with image patches [START_REF] Kumar | What is a good nearest neighbors algorithm for finding similar patches in images?[END_REF].

The VP-tree is characterized by the split value being a hyperball and the split function being the indicator function of this ball. The VP-tree splits the data set according to the distance of each point to a vantage point. The vantage point is one of the data points, chosen according to some criteria. Forests of VP-trees where the vantage point is chosen at random have been shown to have a good retrieval power [OD + 13].

For our experiments, we used a forest of four VP-trees, randomized as in [OD + 13]. We set the size of the bins to 2n, where n is the number of nearest neighbors of the query. The trees are constructed using all patches in the video. For the query, the local refinement area is of size 8 × 8 × 3. We found these parameters to give a reasonable trade-off between computational cost and search accuracy. In the following, we will use the abbreviation VP with local refinement (VPLR) instead of partition tree with local refinement (PTLR) to remind that the partition tree is specifically a VP-tree.

In Figure 2.2 we compare the results obtained with the local search, the "first bin" search, and the proposed VPLR search for the classical test sequence bus1 . The plot shows the Euclidean distances to the n = 300 nearest neighbors averaged over 1000 query patches randomly chosen in the same video. The patches are of size s × s with s = 10, RGB (thus their dimensionality is 300) and their distance is rescaled between 0 and 1 as d(p, q) = pq /(255 √ 3s). The first conclusion is that the local exhaustive search finds worse matches than the approximate global search methods (the search window of the local search is of size 45×45×5). The best method is the VPLR search, performing much better than the basic VP-forest "first bin" approx- imate search. This result shows the effectiveness of the proposed search heuristic. We computed the corresponding plot on the other videos and always found the same qualitative behavior. It is also interesting to visualize the position of the matches found by each method in the spatio-temporal domain of the video. Figure 2.1 presents the position of the nearest patches found for a specific query in the bus video. Note how the matches found by searching globally are organized in trajectories.

For the same parameters, the number of distance computations for each method is 10125 and around 20000 for respectively the local search and the VPLR search for the first step of the algorithm (an equivalent local search region would be close to 63 × 63 × 5).

Experimental results

We evaluated the effect of the global search on the BM3D and NL-Bayes image denoising algorithms. Since the source code of BM3D is not public we use the implementation available in [START_REF] Lebrun | An Analysis and Implementation of the BM3D Image Denoising Method[END_REF]. We adapted it to process image sequences and modify only the patch search as explained in §2.2. For NL-Bayes (referred as NLB in the following), we built our implementation upon [START_REF] Lebrun | Implementation of the "Non-Local Bayes" (NL-Bayes) Image Denoising Algorithm[END_REF] , and modified to limit the rank of the a priori covariance matrix (see §2.2).

For each method we considered the three versions depending on the type of search used in each step: the local (exhaustive in a small spatio-temporal volume) and global (approximative in the entire video) versions use the corresponding search in both denoising steps, and an additional mixed approach, which uses the local search in the first step and the global one in the second. The reason for this will be explained later.

We compared these methods against V-BM3D [DFE07], V-BM4D [START_REF] Maggioni | Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms[END_REF] and SPTWO [START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF], which represent the current state-of-the-art in video denoising.

Regarding the parameters, we considered 2D patches of size 10 × 10 for NL-Bayes and 8 × 8 for BM3D. For the local search we used a window of size 45×45×5 for NL-Bayes and 32×32×5 for BM3D.

The remaining parameters for NL-Bayes are the number of similar patches used in each step n 1 , n 2 and the maximum ranks of the a priori covariance matrix r 1 , r 2 . For BM3D we also needed to specify n 1 , n 2 , in addition to the hard threshold in the first step λ 1 as well as the distance threshold in both steps, τ 1 and τ 2 . The values for λ 1 , τ 1 and τ 2 are the same as the ones in VBM3D. The rest of the parameters were tuned by optimizing the PSNR on a training set consisting of short videos. The optimal parameters depend on the noise level. Table 2.1 synthesizes the different parameters as a function of the noise.

These patches are somewhat larger than the typical sizes used by patch-based methods. The reason for this is that the global search is more sensitive to the noise in the patch distance. Consider for example a noisy flat image. Since the image is flat, the closest neighbors to a patch are the ones with the closest noise pattern. If enough nearest neighbors share the same noise pattern, it will be interpreted as a signal component and will not be filtered out. The global search increases the probability of finding a large number of matches with a similar noise pattern (overfitting to this specific noise pattern). In practice, this becomes an issue for patches in homogeneous regions and high noise values. This problem can be mitigated by involving a larger number of similar patches. An interesting aspect of global search is that it is still possible to find many similar patches even for large patch sizes.

Our quantitative comparison criterion was the PSNR. We first evaluated the effect of the global search for NL-Bayes and BM3D. Tables 2.2 and 2.3 show the gain in PSNR with a global search, on grayscale and color sequences. 2 . In almost all cases the global search performed significantly better than the local for NL-Bayes. This is also true, but to a minor extent for BM3D. For NL-Bayes the highest gain was obtained using the global search in both steps of the denoising algorithm: The average gain between NLB-global and NLB-local is of around 1dB for grayscale sequences and of 1.5dB for color sequences. This gain is consistent across the different noise levels we used in our tests. For BM3D the best alternative is BM3D-mix, which uses the global search only in the second step. The performance of BM3D-global is superior for σ = 10, but drops severely when the noise increases, becoming comparable or even worse than BM3D-local for the highest levels of noise.

A possible reason for this is that BM3D uses a much smaller number of similar patches n than NL-Bayes. As explained before, for patches with low SNR, the global search increases the risk of finding a set of nearest neighbors sharing a similar noise pattern, particularly for small n. Note also that the performance of BM3D is worse than that of NL-Bayes, and in most cases worse than V-BM3D (see Table 2.4). This is because our version of BM3D is an adaptation to video of the one published in [START_REF] Lebrun | An Analysis and Implementation of the BM3D Image Denoising Method[END_REF]. In particular, the search strategies of our BM3D-local and V-BM3D differ, V-BM3D uses the predictive block matching described in §2.2 that is optimized to search in a large temporal window while our search is very local both in space and in time. We compared the performance of NLB-local and NLB-global with the state-of-the-art methods V-BM3D [DFE07], V-BM4D [START_REF] Maggioni | Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms[END_REF] for grayscale (Table 2.4) and color (Table 2.5) videos. The sequences used in these tables were the same as in Tables 2.2 and 2.3. The results of V-BM3D3 and V-BM4D were computed using the authors' implementation. 4 For grayscale sequences and σ = 10, NLB-global has the best performance. On average NLB-global has a PSNR .78dB higher than V-BM4D. When the noise increases, the gap between NLB-global and the V-BMxD methods closes. For most sequences, better results can be obtained with NLB using a larger patch for higher noise levels. This suggest that the problem comes from the distance estimation in these high noise cases.

We also include a comparison with SPTWO in Table 2.6. We computed the result of our algorithm for some of the sequences used in [START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF]. Note that the sequences in Table 2.6 have 30 frames and that the values shown correspond to the PSNR of the central frame of the sequence. The results depends largely on the sequence. In particular SPTWO performs better in tennis, and bus. The fact that the bus sequence has a very fast motion which can be easily estimated might explain the performance gap in favor of motion estimation on this sequence.

Two examples of denoised results are presented in Figures 2.5 and 2.6. Comparing the different results of denoising, we can see that the VPLR search allows a better detail reconstruction. In particular, in Figure 2.5 the numbers of the calendar do not show a blur around them compared to the other methods based on a local search. For the example from grayscale bus, the improvements can be seen mostly for the woman inside the bus, who is more distinct with the global search than with the local search; but also on the ad where most details of the singer's face are better reconstructed.

In section 2.3.3, we briefly discussed the computation complexity (in number of distance computation) of each search method per query. When these searches are integrated into the denoising algorithm, the full computation time (including the construction of the VP-tree) is of the same order of magnitude than the one when using the local search. Nevertheless, NL-Bayes based methods are reasonably slower than V-BM3D and V-BM4D when using the "normal profile".

Conclusions

We studied the performance gain obtained by expanding the local patch search into a global one for patch-based video denoising algorithm. To the best of our knowledge, this is the first time that denoising results using global patch search were reported in videos with hundreds of frames. With the global search the patches found can follow long trajectories in the video, thus fully benefiting from the temporal redundancy of videos.

Our analysis of the most common patch search algorithms showed that an approach based on a global tree structure, more specifically based on a VP-tree, performed very well compared to the local search. Exact global search in the VP-tree is still too costly for the denoising application, which is why we proposed a simple heuristic for efficient approximate search, the VPLR search (VP-tree search with local refinement).

We then applied it to extend BM3D and NL-Bayes, two image denoising algorithms, to video. We obtained a significant boost on the denoising performance. This performance boost is only slightly more costly than a local exhaustive search, including the time spent building the tree thanks to an easy parallelization.

Latest contributions in video denoising advocate for the use of 3D patches as a mechanism to impose temporal consistency in the video [PE09,LF10,MBFE12]. Yet, in this chapter we showed that state-of-the-art results can be obtained with 2D patches, using global search. The results obtained are visually better frame-by-frame, but can suffer from a flickering artifact due to the lack of temporal consistency. This is most noticeable for higher values of noise. Ongoing work focuses on extending the current results to 3D patches and video specific algorithms. One of the current limiting factors associated to the global search is that it increases the risk of matching the noise pattern for patches with low SNR. We were able to alleviate this problem in most cases by using large 2D patches, but this causes problems with random, low-contrasted textures which are better denoised with small patches. 3D patches can reduce the spatial patch size while still keeping accurate distances (same dimension than the 2D patches), and therefore be more appropriate for these types of textures.

The proposed heuristics for approximate global patch search are not limited to the denoising application and are useful for other applications requiring a large number of nearest neighbors. An example of application to anomaly detection is presented in Chapter 9.

A recursive video denoising method: NL-Kalman

In this chapter we propose a new recursive video denoising method with high performance.

The method is recursive and uses only the current frame and the previous denoised one. It considers the video as a set of overlapping temporal patch trajectories. Following a Bayesian approach each trajectory is modeled as linear dynamic Gaussian model and denoised by a Kalman filter. To estimate its parameters, similar patches are grouped and their trajectories are considered as sharing the same model parameters. The filtering is mainly temporal; nonlocal spatial similarity is only used to estimate the parameters. This temporally causal method obtains results comparable (in terms of PSNR and SSIM) to methods using several frames per frame denoised, but with a higher temporal consistency. This work has been published in [START_REF] Ehret | Non-local kalman: A recursive video denoising algorithm[END_REF].

Introduction

There are two current trends in video denoising. The first one aims at producing the best video denoising possible whatever the computation required, while the second trend focuses on producing the fastest causal video denoising algorithm with the goal of real-time processing.

In terms of output quality the state of the art is achieved in part by patch-based methods [DFE07, MBFE12, AM18a, EAM17, BLM16, WLPB17]. As we have previously seen in Chapter 1, they exploit the self-similarity of natural images and videos, namely that most patches have several similar patches around them (spatially and temporally). Each patch is denoised using its similar patches, which are searched for in a region around it. The search region generally is a spatio-temporal cube, but more sophisticated search strategies have also been used. Because of the use of such neighborhoods these methods are called non-local. While these algorithms perform very well, they often are unpractical: they use a frame's past and future and therefore can only be used off-line. Because of their complexity they are unfit for high resolution video processing.

Fast algorithms rely on much simpler principles which can be implemented efficiently on GPUs or FPGAs. For example [PPR + 17] relies on a bilateral filter and a Kalman filter to produce a real-time video denoising algorithm. A recursive version of the non-local means algorithm is proposed in [START_REF] Redha | Recursive non-local means filter for video denoising[END_REF]. These methods use simple denoising strategies yet generally result in poor denoising quality for high noise levels.

Convolutional networks have been successfully applied to image denoising, and also to other video processing tasks such as deblurring [SDW + 17] or video synthesis. Their application to video denoising has been limited so far. In [START_REF] Chen | Deep rnns for video denoising[END_REF] a recurrent architecture is proposed, but the results are quite below the state of the art. Recently [START_REF] Godard | Deep burst denoising[END_REF] focused on the related problem of image burst denoising reporting very good results.

In this chapter we present a video denoising method that tries to close the gap between highquality but slow methods and those efficient but of lower quality. It is much faster than most state-of-the art algorithms (yet not real-time) but in contrast to them it is causal temporally and recursive, i.e. it uses only the current frame and the previous denoised frame. In spite of these strong design constraints it achieves a state-of-the-art performance in image quality.

Proposed Method

In the following we denote by u a clean video which has been contaminated by an additive Gaussian white noise n of (known) standard deviation σ, i.e. only v = u + n is observed. Even though the additive white Gaussian model seems simplistic, it is enough to also process more realistic Poisson noise. Indeed, Poisson noise can be reduced to nearly white Gaussian by a variance stabilizing transform [START_REF] Zhang | Multiscale variancestabilizing transform for mixed-poisson-gaussian processes and its applications in bioimaging[END_REF]. The video u, respectively v, is made of f frames indexed between 0 and f -1; the frames are denoted by u t , respectively v t , with t ∈ 0, f -1 . We will work with two dimensional square patches of size s × s, a patch will be denoted by a bold lower case letter such as p, considered as a vector of dimension s 2 .

Framework

Our method works by temporal filtering along patch trajectories using a Kalman filter associated to each trajectory. The Kalman filter requires for its operation to keep track of the state covariance matrix. The state in our case is the clean patch that we want to estimate (arranged as a vector of s 2 components), and the state covariance is a s 2 × s 2 matrix. We use a simple dynamic model for the temporal evolution of the patch trajectories, which requires the estimation of a single parameter at each time step: the state transition covariance matrix. It models how a (clean) patch can vary from one frame to the next. Estimating the state transition covariance matrix is in general a difficult problem. We resort to non-locality for obtaining a reliable estimate. To that aim, we group trajectories based on the similarity of the first patch, and assume that these trajectories share the same model. The covariance matrix is obtained from the statistics of the trajectories in the group. This assumption also allows us to reduce the memory consumption of the method, since all the trajectories in the group, because they share the same transition covariance, also share the state covariance. Because there is still a small high frequency residual noise after the NL-Kalman pass, the multiscale DCT denoising algorithm [START_REF] Pierazzo | Multi-Scale DCT Denoising[END_REF] is applied to each frame as a post-processing.

The definition of patch trajectories, and their computation from an optical flow is explained in Section 3.2.2. Patch trajectories can be terminated if an occlusion happens (or more generally due to a registration error). This is explained in section Section 3.2.5. The spatial denoising algorithm is responsible for initializing the groups of trajectories, and is explained in Section 3.2.3. Finally, the key of our contribution, namely the non-local Kalman filters running on groups of trajectories, is explained in Section 3.2.4.

Patch trajectories

A patch trajectory is temporal sequence of s×s patches q 0 , q 1 , . . . , q t extracted from consecutive frames in the video. The centers of these patches follow a motion trajectory, which is estimated using an optical flow algorithm. Suppose we know the trajectory of a patch until frame t -1, q 0 , q 1 , . . . , q t-1 . The next patch in the trajectory at frame t is computed by tracing the optical flow between frames t and t + 1. We assume that all the pixels in the patch move with the same optical flow as the patch center. This is a crude approximation but it allows for a simpler dynamic model.

We use an optical flow to compute the patch trajectories because it is dense and is not restricted to any particular type of motion (as opposed to e.g. global parametric models). In particular, we used the implementation of [START_REF] Sánchez Pérez | TV-L1 Optical Flow Estimation[END_REF] of the TV-L1 optical flow [START_REF] Zach | A duality based approach for realtime tv-l 1 optical flow[END_REF]. Any other motion model can be used, as long as it provides a dense set of correspondences. Optical flow is less robust to noise than global parametric models. To gain robustness to noise (and speed), we compute it on a downscaled version of the frames, similar to [LYT + 14].

Spatial denoising and starting groups trajectories

The spatial denoising is used when no temporal information from the previous frame is available (e.g. in a dis-occluded area). The goal here is not only to denoise these areas, but also initialize the groups of trajectories for the temporal denoising of these areas in the future frames. We use a single step of the NL-Bayes denoising algorithm [START_REF] Lebrun | A Nonlocal Bayesian Image Denoising Algorithm[END_REF], which has the benefit of computing groups similar patches and their covariance matrices (needed for the Kalman filters) as part of the denoising.

Given a patch q of the noisy frame v t , and the corresponding unknown patch p of the clean frame u t , the following Gaussian linear model of p and q is assumed: P(p) = N (µ, C) and P(q|p) = N (p, σI). Once the mean patch µ and the covariance matrix C have been estimated from the noisy video, the MAP estimate p given a noisy patch q is obtained as in [START_REF] Lebrun | A Nonlocal Bayesian Image Denoising Algorithm[END_REF]:

p = µ + C(C + σ 2 I) -1 (q -µ).
The parameters of the a priori model are learned from the noisy frame. For each noisy patch q, N similar patches are selected from the frame. Let q i , i = 1, . . . , N be the set of patches similar to q (with q 1 = q). The estimates for µ and C are given by μ = 1 N N i=1 q i and Ĉ = 1 N N i=1 q i q T i -σ 2 I, where q i = q i -μ are the centered patches. We call group of trajectories such a set of patches with their C associated. The idea is to follow the evolution of that group of patches by tracking their respective trajectories. We assume a common destiny for all these patches and so we update C jointly for all these patches along the trajectories. This recursive step is presented in the next section. corresponds to the proposed method using an optical flow computed on the clean sequence. Results with a star were computed using the binary provided by the author. Best results are in bold (results from the oracle are excluded). VBM3D has the lowest running times, followed by NL-Kalman and finally VBM4D. Other state-of-the-art methods like SPTWO [START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF] and video NL-Bayes [AM18a] can achieve better results, but with a significantly higher running time. See text for details.

Temporal filtering

We propose a recursive temporal filtering. We assume available a set of groups of trajectories estimated from the previous frame (either from the temporal filtering or the spatial initialization) such that the union of all trajectories in the groups covers that frame. To denoise the frame t, we loop on the groups updating the parameters of the Kalman filter and operating it to estimate the clean patches of the trajectories in the group at frame t.

Let G = {p t-1,1 , . . . , pt-1,N } be a group of trajectories. For each patch pt-1,i we have a noisy observation q t,i . The registration test compares pt-1,i and q t,i and determines if the registration is correct or not. This is detailed in Section 3.2.5. The trajectories that do not pass this test are terminated and removed from the group. In the following we assume that the N trajectories in the group passed the test. For such patches, we have a prediction, namely the previous denoised patch in the trajectory.

We assume that the evolution of the patch along a trajectory follows a simple linear dynamic Gaussian model, described as follows:

p t+1,i = p t,i + w t,i with w t,i ∼ N (0, C t ) (3.1) q t,i = p t,i + n t,i with n t,i ∼ N (0, σ 2 I). (3.2)
Here w t,i ∼ N (0, C t ) is the process noise, modelling the variations of a patch from one frame to the next. It depends on t because we assume that patches can evolve (slowly) in time. The only model parameters that need to be estimated are the state transition covariances C t associated to the group. It is easy to verify that

E{(q t,i -q t-1,i )(q t,i -q t-1,i ) T } = C t + 2σ 2 I. (3.3)
By assuming that the patches in a group are independent realizations of the same dynamic model, we can estimate the expectation as the sample covariance matrix of the innovation vectors q t,i - From top to bottom: Crop from the sequence crowd_run and pedestrian_area. Overall the proposed method preserves more details as can be seen both in the tree and in the text. Results best viewed zoomed. q t-1,i of the trajectories in the group. We thus estimate C t as the following spatio-temporal average:

C t = βC t-1 + (1 -β) • N i=1 (q t,i -q t-1,i )(q t,i -q t-1,i ) T N -1 -2σ 2 I , ( 3.4)
To increase the temporal stability of the estimate, we combine the non-local average over the group with the previous estimate C t-1 , with a forgetting factor β ∈ [0, 1]. The initial C 0 is set as the covariance of the spatial denoising. We estimate the patches and update their covariance with the Kalman filter equations [START_REF] Emil | A new approach to linear filtering and prediction problems[END_REF]:

Predicted estimate cov.

P t = P t-1 + C t (3.5) Optimal Kalman gain K t = P t (P t + σ 2 I) -1 (3.6) Update state pt,i = pt-1,i + K t (q t -pt-1,i ) (3.7) Updated estimate cov. P t = P t -K t (P t + σ 2 I)K T t (3.8)
Here P t is the state covariance matrix, it needs to be stored with the group of patches (as the state transition covariance C t ). The dynamic model is initialized using the result of the spatial denoising on the patch group. In particular the initial state covariance matrix is taken as the covariance computed by the NL-Bayes spatial denoising algorithm.

The proposed model can be interpreted as a dynamical version of the video NL-Bayes denoising method [START_REF] Arias | Video denoising via empirical bayesian estimation of space-time patches[END_REF]. Their similar 3D patches (of size around 7 × 7 × 2) are assumed to be normally distributed. The parameters of these Gaussian distributions are estimated from the set of similar patches, gathered from past and future frames. The estimated covariance matrix represents the modes of variation of the population from the estimated mean patch. Instead our Gaussian dynamical models represent temporal variations along patch trajectories.

Registration quality assessment

We now address the registration quality assessment, to detect occlusions and errors in the optical flow. The quality must be assessed for each patch trajectory individually. The idea is to compare the denoised patch (the prediction in the Kalman filter) to the new noisy observation. Let q t be the patch candidate and p t-1 the result of the denoising for the corresponding patch at the previous frame. The assumption underlying the registration quality assessment is that q t is a noisy version of p t-1 (Eq. (3.2)). Therefore

p-q 2 2 σ 2 ∼ χ 2 (d).
Our goal is to reject patches that are too different, as the dynamic system can only deal with small changes. The optimal test to assess the quality of the matching corresponds to p-q 2 2 σ 2 ≥ c. In practice it is better to also take into account the temporal variation of patches but this will be disregarded here for simplicity.

In order to choose the best possible c, one can use a statistical approach like in [DMM07, VGJMR10, PGvG12, LGvGRM14, MS04, PGvGO13] based on a number of false alarm (NFA). We would like the patch p to be rejected if and only if we are confident enough that it is not a good match and not because of the noise i.e. N P p-q 2 2 σ 2 ≥ c ≤ ε where ε corresponds to the number of false matches that can be accepted and N to the number of time a patch is added to trajectory. This leads to c ≥ G -1 1 -ε N where G -1 the inverse cdf of a chi square distribution of dimension d. A sound value for N is the number of pixels of a frame. Indeed, fixing = 1 amounts then to admit at most one false matching on average per frame.

Experiments

We compared the performance of our algorithm against the state-of-the-art denoising algorithms VBM3D and VBM4D. Our causal recursive method requires a couple of frames to warm up. Therefore we remove the first five frames in the computation of the PSNR and SSIM to get a fair comparison. Figure 3.1 shows the quick improvement of the denoising quality in the first three frames used as warm up. Table 3.1 presents denoising results on 1920 × 1080 sequences from Derf's video database 1 that have been converted to gray by averaging the RGB channels and downscaled by two so as to ensure that they contain little noise. A Gaussian blur was applied before downscaling to avoid aliasing.

Overall the proposed method performs better than VBM3D and VBM4D in SSIM but shows a lower PSNR. The main limitation of NL-Kalman stems from the optical flow. Estimating the optical flow from noisy data is challenging. This causes miss-registration errors which explains the observed drop in PSNR performance. To evaluate this we compare the denoising results obtained with optical flow computed on the noisy data to those obtained from ground truth optical flow. In spite of this, the result of NL-Kalman shows a much higher visual quality, in terms of temporal consistency and in the amount of recovered details. This explains why the difference between the SSIM measure, where NL-Kalman leads, and PSNR, where the average difference is about 0.7db in favor of VBM4D. A comparable result in terms of temporal consistency is attained with SPTWO [BLM16], a more complex method using 10 frames for each frame denoised. SPTWO is also based on the optical flow, but is not recursive. To denoise each frame it computes the optical flow between the target frame and n neighboring frames (n/2 past plus n/2 future frames). We computed SPTWO with n = 8 for σ = 30 and obtained an average PSNR of 32.45 (SSIM .6114).

Among the algorithms compared, the faster one is VBM3D. Our current implementation in non-optimized C++ code is between two to three times slower than VBM3D. The main reason for this is that the number of groups grows as time evolves, since new groups are created for covering patches in dis-occluded areas. We are currently working on faster versions of the algorithm by limiting the number of groups.

Conclusion

We presented a novel patch-based video denoising method and showed that it is competitive with the state of the art. It is based on temporal filtering along trajectories of patches. The temporal filtering is carried out by Kalman filters whose parameters are estimated non-locally. The resulting method is recursive, as to denoise one frame it uses solely information from the previous frame. Results show a performance comparable to more complex patch-based methods that use around ten frames to denoise each single frame. The main limitation of the method is that it relies heavily on the optical flow. Slight errors in registration cause a drop in PSNR, even if visual quality of the result is much superior to the one of related algorithms, both in terms of details and temporal consistency. This work also shows the limits of the PSNR and SSIM as quantitative measures for video quality.

4 Fast patch-based denoising on GPU

We have seen in the previous chapters that denoising is an essential part of any image or video processing pipeline. Unfortunately, due to time processing constraints, many pipelines do not consider the use of modern denoisers. These algorithms have only CPU implementations or suboptimal GPU implementations. In this chapter, we propose a new efficient GPU implementation of NL-means and BM3D, and, to our knowledge, the first GPU implementation of the video denoising algorithm VBM3D. The performance of these implementations enable their use in real-time scenarios. This work has been published in [DE].

Introduction

As we have seen in Chapters 1, 2 and 3, there are two current trends in video denoising. The first one aims at producing the best video denoising possible whatever the computation required (Chapters 1 and 2), while the second trend focuses on producing the fastest causal video denoising algorithm with possibly the goal of real-time processing (3).

An interesting middle ground that we will present in this chapter is to produce optimized implementations of the well performing denoising methods for optimized hardware such as GPUs or FPGAs. This has often been done for image denoising with optimized implementations of NLmeans [START_REF] Márques | Implementation of non local means filter in gpus[END_REF][START_REF] Palhano | Real time ultrasound image denoising[END_REF] and BM3D [START_REF] Honzátko | Accelerating block-matching and 3d filtering method for image denoising on gpus[END_REF][START_REF] Wang | Accelerating block-matching and 3d filtering-based image denoising algorithm on fpgas[END_REF]. However, even though performance is even more crucial for video, little work has been done to optimize video denoising methods. The only exception being for video versions of NL-means [GLA + 10].

In this chapter, we propose real-time GPU implementations of an improved patchwise NLmeans and BM3D. Both implementations outperform all previous proposed GPU implementations, without diminishing the quality of the results. We also use the same backbone to get the first optimized GPU implementation of VBM3D. The performance of this implementation is sufficient for real-time video filtering. This is achieved by regrouping all the filtering operations (fetching the patch data, the data transpositions, the 1D filterings and the thresholding) into one kernel without requiring an intermediate buffer for the patches being processed. This significantly reduces the use of memory bandwidth. Moreover, all memory accesses are designed so as to reduce bandwidth and benefit from the cache.

The chapter is organized as follows. Section 4.2 presents the studied methods. Section 4.3 presents the specificity of GPU implementations. Section 4.4 presents the efficient GPU implementations for the methods presented in Section 4.2. The efficiency of the proposed implementations are measured in Section 4.5. Finally Section 4.6 concludes.

Architecture of the implemented Non-local denoising algorithms

Both NL-means and BM3D (and their video extensions) are based on a self-similarity principle. This means that their architectures are relatively similar; built around a patch search step as well as a patch group processing step. We present the different methods in detail in Sections 4.2.1, 4.2.2 and 4.2.3 as well as the different implementation options.

NL-means and its extension to video

Original NL-means. NL-means [START_REF] Buades | A non-local algorithm for image denoising[END_REF] is a popular image and video denoising algorithm, due to its simplicity and speed. It introduced the self-similarity hypothesis: for a patch (a small region of the image) of a natural image, many similar patches can be found in its neighborhood. For noisy images, these similar patches will have different noise realizations and therefore their information can be combined to estimate the restored information. The original NL-means works as follows: for each position (x, y) of an image u, consider the corresponding reference patch P (x, y) centered at this position. The value of the estimated image û at position (x, y) is computed as a weighted average of all patches in the small local neighborhood N (x, y) centered on (x, y). While the self-similarity hypothesis says that similar patches should be found, not all patches in the local region are similar. The weights are therefore used to reduce the impact of patches that are too different. The original NL-means suggested the weighted average presented in Equation (4.1) where the only parameter h is here to take into account the noise.

û(x, y) = 1 C x,y x ,y ∈N (x,y) w x,y (x , y )u(x , y ). (4.1) with w x,y (x , y ) = exp - P (x, y) -P (x , y ) 2 2 h 2 (4.2)
and C x,y =

x ,y ∈N (x,y)

w x,y (x , y ) (4.3)
Improved NL-means. In this chapter, an alternative version of NL-means is implemented. The alternative version was preferred since it is more efficient (it requires less computations) while still achieving denoising on par with the original implementation. The improved version follows the same global structure as the NL-means presented in the previous paragraph. The alternative relies on four major differences. First, we use Equation (4.4) instead of Equation (4.2). This improvement proposed in [START_REF] Buades | Non-local means denoising[END_REF] was shown to reduce a matching noise bias in [START_REF] Frosio | Statistical nearest neighbors for image denoising[END_REF].

w x,y (x , y ) = exp - P (x, y) -P (x , y ) 2 2 -2σ 2 + h 2 (4.4)
We decided to limit the number of patches used in Equation (4.1). Indeed instead of using all patches from the local search region N (x, y), only use a subset comprised of the n best nearest neighbors is used. This is possible since most patches in the search contribute very little in Equation (4.1). Very often only a few patches have a distance small enough to contribute. The exception of the flat regions is mentioned later. This choice is principally for computation reasons as it speeds up the method as shown in [START_REF] Mahmoudi | Fast image and video denoising via nonlocal means of similar neighborhoods[END_REF] and [START_REF] Coupé | Fast non local means denoising for 3d mr images[END_REF]. However it can also improve the quality of the results, acting as a regularizer as shown in [START_REF] Gilboa | Nonlocal linear image regularization and supervised segmentation[END_REF] and [START_REF] Kervrann | Bayesian non-local means filter, image redundancy and adaptive dictionaries for noise removal[END_REF]. We also added a "flat patch trick" similar to NL-Bayes [START_REF] Lebrun | A Nonlocal Bayesian Image Denoising Algorithm[END_REF]. For that, a simple test on the variance of the n patches found is added. A flat region is detected when the variance of the pixels from these patches is too close to the variance of the noise (see Equation (4.5) with β = 1.05).

V ar

[{P i | i ∈ {1, . . . , n}}] < βσ 2 (4.5)
In that case we simply average the contribution of all pixels to produce the estimated patch (all pixels of the estimated patch have the same value). This means that n × p × p contributions are averaged instead of only n contributions where p is the width of the patch. This improves the quality of the denoising in flat regions, especially when the number of patches used is small. The last improvement is the use of a patchwise NL-means as proposed in [START_REF] Buades | Non-local means denoising[END_REF]. In the patchwise NL-means variant, the entire patch is denoised by averaging and the entire estimated patch is aggregated instead of just its center pixel. However, instead of using the uniform weighting of [START_REF] Buades | Non-local means denoising[END_REF] for the aggregation, we use bilinear weighting. This means that pixels near the center ot the patch will have a greater impact during aggregation. The advantage of the patchwise NL-means variant is that an aggregation step can be introduced to skip some pixels like in BM3D [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF] and NL-Bayes [START_REF] Lebrun | A Nonlocal Bayesian Image Denoising Algorithm[END_REF] and thus limits the amount of computation. The final estimate of these pixels comes from the aggregation of the overlapping results from the other patches. The improved version of NL-means is described in Algorithm 10 (also shown in 4.1) and the differences are summarized in Table 4.1. This is not the first work to try to improve NL-means running time or improve the denoising performance. Examples of acceleration are [START_REF] Brox | Efficient nonlocal means for denoising of textural patterns[END_REF] which uses a cluster tree for the patch search and [WGY + 06] which use FFT and the Summed Squares Image for the filter computation. Examples of denoising performance improvements are [START_REF] Kervrann | Bayesian non-local means filter, image redundancy and adaptive dictionaries for noise removal[END_REF] which uses a Bayesian model to replace the NL-means weights. Instead of using all the patches in the window, a locally adapted dictionary is extracted locally from the image in a first pass, and from a first denoising result -or oracle -in a second, [START_REF] Duval | On the parameter choice for the non-local means[END_REF] proposes a local adaptation of the smoothing parameter h using a method based on SURE, [START_REF] Sutour | Adaptive regularization of the nlmeans: Application to image and video denoising[END_REF] improves the output of NL-means, especially on regions with few good neighbors, by minimizing a global cost function involving the NL-means weights, the NL-means output and the TV norm, [START_REF] Jin | Nonlocal means and optimal weights for noise removal[END_REF] replaces the NL-means weights with triangular kernels and a scheme to automatically adapt the smoothing parameter. The authors claim that their proposed kernels can be compared to the one proposed in [START_REF] Buades | Non-local means denoising[END_REF] which we use. However all these works focus on CPU implementations. Indeed their modifications are not adapted for

BM3D by Dabov et al.

The denoising principle of BM3D exploits the redundancy of similar patches. Groups of similar 2D patches are assembled in a 3D stack. A separable 3D orthonormal transform is applied to this stack. The stack is denoised by applying a shrinkage operator to the coefficients in a transformed domain. The transform is selected to reveal sparsity in the transformed domain. Most DCT patch coefficients are for example negligible in natural images. However the transform being a patch isometry, the noise remains spread evenly among all DCT coefficients. The algorithm follows four basic steps:

1. Search for similar patches in the image and group them in 3D stacks, 2. Apply a 3D linear domain transform to the 3D blocks, 3. Shrink the transformed coefficients, 4. Apply the inverse transform, 5. Aggregate the resulting patches in the image.

The construction of the estimated image is done by aggregation, i.e. by combining the results of all the different estimations. This principle is applied twice, once to compute a basic estimate and a second time, using the basic estimate as a guide, for a final estimation. In the second stage, this process is indeed repeated but uses the denoised patches of the first "basic" step to find similar patches. Furthermore, transform thresholding is also replaced by a Wiener denoising using the denoised patch as oracle. The method is synthesized in Figure 4.2. In short, we implement exactly the same method as the one presented in [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF].

VBM3D by Dabov et al.

As we have seen in Chapter 1, VBM3D is a straightforward extension of BM3D to video. The only major difference lies in the patch search. The method was synthesized in Figure 1.1. Since the 7 × 7 patch size of the second step is difficult to implement efficiently on a GPU, we decided to use 8 × 8. We checked using the implementation from Chapter 1, that this choice incurs in no significant performance loss. See Table 4.2. This is the only difference compared to the method proposed in [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF].

An introduction to GPU architecture

To understand some of the implementation choices in Section 4.4, some knowledge about GPU computing workloads is required. We shall thus give an overview of the main particularities and challenges of GPU programming. In this section, we define a coherent hardware terminology.

GPUs and OpenCL for highly parallel workloads. Modern GPUs have a highly superior computing power compared to modern CPUs; they are highly multithreaded. There can be 10 5 to 10 6 threads running at the same time. While parallelization of many data processing algorithms is generally straightforward on CPU since CPUs' cores are fairly independent, GPUs need more care to take advantage of their highly parallelized architecture. In OpenCL, the programmer writes kernels, which are functions executing on the GPU. All assigned threads execute the same kernel code but start with a different work-item index. This leads each kernel to do different portions of the work that needs to be parallelized. Alongside kernels, an OpenCL program also needs CPU code to manage the GPU memory and define when and with which arguments these kernels should be executed.

The programmer has some control on how indices are assigned to GPU threads. First, threads are assigned linearly. This enables optimizations so to have consecutive threads access consecutive memory regions. Second, the programmer can define a workgroup size (called local size). The global size is divided into regions of size defined by the local size. Each region is assigned to a workgroup. The main interest of a workgroup is that its threads have access to a common local shared memory only visible to them. This memory enables fast communications between threads. It can be used to store common computations or to cache memory accesses.

A single instruction multiple threads (SIMT) model When assigning indices to threads, the hardware groups consecutive indices of a same workgroup into a warp. Each thread inside a warp will execute in lockstep. Thus every thread will execute the same instruction at the same time. For example their memory accesses will be simultaneous. This knowledge can be used to reduce the cost of memory accesses. However this model also implies that if several threads need to take different code paths -this phenomenon is called thread divergence -all code paths taken need to be executed by the warp, disabling the threads that should not take this code path.

Memory accesses with GPUs Memory accesses are affected by both latency and bandwidth. Latency is the time taken for an operation to complete. GPUs have two ways of hiding latency. First the code instructions can be organized to have non-dependent computations inserted between a memory access and the code requiring its result. In this case the hardware only needs to wait if the memory access has not finished when the code requiring its results needs to be executed. Second several warps are scheduled on the same computing resources. When a warp has to wait, other warps can be executed.

The bandwidth is the maximum amount of data the hardware can read or write from the memory per second. To maximize the usage of available bandwidth, optimized memory access patterns are required. Any access to read memory at a given location actually loads a batch of consecutive elements (called cache line).

Memory accesses are cached to improve latency and bandwidth. Reading the same data again while it is still in cache enables faster access. However, GPU caches are very small relative to the number of threads executing. To compensate for this, GPU threads have access to a significant number of registers, which are local storage only accessible by a given thread.

More details on GPUs can be found in technical documents such as [Jun15, AMD15, NVI09].

Implementation details

4.4.1 Patch search with a large, 2D or 3D, fixed window

In this section we shall describe how to implement an efficient patch search in a predefined 2D (or 3D in the case of video) search window. The problem can be described as such: Given an image subgrid with a pixel spacing of step pixels, we want to compute the n best neighbors for each position (the distance between patches used can be L 1 or L 2 for example). The patch is of size p × p and the w × w search window is centered on the reference patch. In the case of a video, a temporal depth is added to the window. Typical values for the parameters are p = 8, w = 21, n in {8, 16, 32} and step in {3, 4}.

Design choices

Implementing this algorithm raises several challenges. Memory accesses must be optimized to make optimal use of the cache. Indeed some computation is common for neighboring patches when p > step and therefore can be shared. Writing in memory all the computed distances would be heavy and thus only a table of the best n patches must be maintained during patch comparison.

Only the positions (and if needed the distances) of the n best patches are written at the end.

For the best efficiency, tables of best patch positions and their weights must be stored in GPU registers. This leads to solutions where one thread will manage all the best neighbors for a given location. This also discourages solutions where a given thread would compute, for a given window offset, the distances between several reference patches and their compared patches. Or solutions where a given thread would handle all distances for a compared patch rather than a reference patch. Thus in the following a given thread will always track distances for a fixed reference position.

When a given thread computes the distances between a reference patch and patches in a window, a lot of data is shared between the computations: the reference patch data is used for all computations and most of the data is in common between a compared patch and its neighbor. The distances must however be recomputed from scratch as different pixels are compared. Because of the restricted register space, storing in registers the equivalent of two p × p patches per thread may not be possible or be desirable. This encourages spreading distance computations for a given reference patch among several threads. These threads would store a part of the reference patch as well as a part of compared patches and reuse them across computation.

Efficient nearest neighbor table

Since writing all patch distances in memory would not only require a lot of memory space, but also bandwidth, we want to only store the final best n neighbors. This implies at any point of the algorithm to remember the best n neighbors seen so far (their positions and their distances), and forget the other positions and distances.

On many hardware, accessing an array with a dynamic index (an index unknown at compilation time) requires storing the array in memory rather than in registers. Even if the array ends up staying in cache, the accesses would suffer from latency and bandwidth limitations. Thus to prevent suboptimal performance, the table accesses must always be done with a fixed index. When accessing the array with an index depending on a loop counter, for example, the loop must be unrolled (compiler hints enable that operation). Unrolling the loop means the generated code does not contain the loop jump. Instead the loop content is repeated for each loop index value, thus the array index is known during compilation. Small arrays with only fixed indices accesses can have their content assigned to registers instead of memory.

We propose two alternative algorithms to keep the n best distances and positions: Algorithms 11 and 12. Keeping an ordered table (Algorithm 11) has several advantages over an unordered table (Algorithm 12): The maximum distance is known directly and does not need to be recomputed, and thread divergence is lower. Indeed in Algorithm 12, threads are more likely to insert their elements at different indices than in Algorithm 11.

We found Algorithm 11 to be slightly faster indeed in our tests on a NVIDIA GPU. However, in our tests on INTEL and AMD GPUs, the performance was significantly lower due to the com-

Algorithm 11: Keeping an ordered table of distances and positions

Input: New position pos and distance dist, two tables Positions and Distances of length n each.

1 if dist < Distances[n-1] then 2 for i from n-1 to 1 do 3 insert ← Distances[i-1] ≤ dist 4 Positions[i] ← pos if insert else Positions[i-1] 5 Distances[i] ← dist if insert else Distances[i-1] 6 quit function if insert 7 Positions[0] ← pos 8 Distances[0] ← dist

Naïve algorithm

In this naïve algorithm, each thread is assigned a different reference patch for which to compute all the distances and maintain the best n neighbors. No particular effort is spent to encourage the compiler to use registers as a cache to reduce the number of memory accesses.

Threads of the same workgroup are assigned to reference patches on the same row. Thus all memory accesses are done on the same image rows, to minimize the number of cache lines per memory accesses in a warp, and more generally improve the cache usage in our scenario. The efficient nearest neighbor table presented above is used.

Convolution algorithm

This algorithm conceptually implements distance computations as separable convolutions while avoiding storing distances. Threads of a workgroup get assigned consecutive positions on a row. For each window offset, each thread will compute the distance between the p pixels of the column at its reference position and at the compared position. This partial result is written in local shared memory. Each thread then reads the results of neighboring threads in order to determine the distance for the whole current p × p patch. Only threads with positions on the image subgrid maintain the best n neighbors. For step = 1, this algorithm is the best performing among the proposed algorithms, due to its very efficient work-sharing and memory pattern. However for higher steps, some computation resources are wasted as a lot of threads do not maintain any nearest neighbor table, an operation with non-negligible cost (due to the phenomenon described in Section 4.3). Note that no threads are assigned to rows which do not intersect with the image subgrid of reference patches.

As p is small enough, the reference pixels can be stored in registers and not reloaded. As for the pixels of the compared patches, on systems with fast local shared memory, memory accesses can be sped up by storing once (and reusing multiple times) the data required for all the comparisons for a given row of the search window.

Square algorithm

This algorithm is optimized for the case p = 8 and step = 4. In that specific case, the distances between a reference patch and its compared patches can be split into four distances of 4 × 4 smaller patches, and these sub-distances are exactly shared among four reference patches. Thus this algorithm proposes to divide the image subgrid into 16 × 16 squares. Each square is assigned to a workgroup (thus workgroups of 256 threads). Each thread tracks the distances for a given reference patch, but only needs to compute the distances for the top 4 × 4 region of its reference patch. Each thread writes its intermediate result to local shared memory. They can then retrieve the distance for the entire 8 × 8 patch by reading the results from three other threads. Naturally, threads at the bottom and right borders of the square workgroup cannot compute the full distances, as no thread computed the required remaining distances outside the square. Thus the workgroup computes the neighbors for a 15×15 region of the subgrid. Some overlapping is required to cover the whole image.

This work subdivision reduces efficiently memory accesses (the reference 4 × 4 patch can be stored in registers, and the memory accessed for the compared patch can be reused for the next patch and kept in registers). Moreover, threads can use memory commands to load four consecutive pixels per call, which reduces the memory instruction count.

Column computations algorithm

This algorithm is an extension of the convolution algorithm, adapting ideas from the square algorithm. It is less efficient than the square algorithm due to p not being as neatly divided by step, but is more efficient than the convolution algorithm when step > 1. Similarly to the convolution algorithm, a group of threads is assigned on a row to compute distances between a reference column and a compared column each. However, contrarily to the convolution algorithm, each thread computes not one, but k + 1 distances. The columns are composed of p + k * step pixels, and the computed distances are the k + 1 partial distances for the intersecting reference patches on the subgrid. The partial results are written in local shared memory and some threads (possibly the same ones) access the results to track the distances for a given reference patch. While for the convolution algorithm, the number of reference patches tracked per thread was low (on average 1/step), in this algorithm k can be controlled to have almost one patch tracked per thread.

Optionally, the partial distance computations and the neighbor tracking can be done on different threads (from separate warps preferably to not waste computational resources), and thus the total register need can be reduced per thread. Indeed if threads do either the partial distance computations or the best distances tracking, the registers used to keep the best weights and distances can be the same as the ones used to store the column reference pixels for example. While for some algorithms and hardware, it is advised to use a low number of registers to have better memory latency reduction, in our case performance was not affected. We believe latency was not an issue for our algorithm.

VBM3D's patch search

In this section we shall describe how to implement an efficient video patch search on the model of VBM3D's patch search. This patch search is similar to the one of Section 4.4.1. However this time the search is performed in multiple frames of the video. First, the q most similar patches are found in each frame. Then, the n best among these neighbors are kept for the rest of the processing. The reference frame (frame containing the reference patch) has a search window of size w 1 × w 1 centered on the reference patch while the other frames have search regions that are the union of q search windows of size w 2 × w 2 centered on the best neighbors of the previous frame. Each time the reference patch is of size p × p and is sampled on a subgrid of pixel spacing of step pixels. Default VBM3D parameters are p = 8, w 1 = 7, w 2 = 5, q = 2, n = 8, step = 6 for the first pass and step = 4 for the second.

Design choices

The challenge of this patch search compared to the one of Section 4.4.1 is that, except for the reference frame, the search windows are shifted by an offset that depends on the patch. While threads treating patches on the same row of the subgrid would access the same rows at the same time for Section 4.4.1, in this section different rows would be accessed because of the patchdependent shifts. Moreover a step of 6 means there is quite a gap between the areas accessed by the threads, thus more cache lines would be accessed if using such a partition.

To keep efficient memory access patterns, several threads can be assigned to the same reference patch. Threads can either handle comparisons for different parts of the search windows or the reference patch can be separated into subpatches where each thread is assigned the comparisons of one subpatch. In both cases, the partial results computed from each thread must be communicated to a main thread that will combine them.

Naive algorithm

For this naive algorithm, each reference patch is assigned one thread that computes all distances and determines the n best neighbors with the required temporal constraints (a maximum of q neighbors per frame). Except for the differences in handling the search region, it is essentially similar to the naive algorithm of Section 4.4.1.

Subdividing the test window

For this algorithm, t threads are assigned per reference patch. For example t can be set to w 1 or w 2 . The search windows are divided into columns, and each thread searches for the q nearest neighbors in its assigned column(s). One thread combines the results of the other threads using local shared memory. This work partition optimizes the memory access pattern. Indeed all t threads will access neighboring data.

However, as said in Section 4.4.1, due to the restricted register space, storing the equivalent of two p × p patches in registers may not be possible or desirable. Thanks to the subdivision in columns, the number of distances computed per thread is small enough so that distances can be computed in several steps by storing partial results in registers. In order to do that, the patches are divided into l blocks (composed of columns of the patch). We compute all the distances for a given block, then the next, until the whole distances have been computed. With this method, the partial patch data is small enough to be kept in registers during the computation of the partial distances for a given block. This optimization reduces the number of memory operations significantly. Note that there is no intermediate buffer for the processing of the patches.

Subdividing the tested patches

For this algorithm, p threads are assigned per reference patch. Each thread computes all partial distances for one column of the patch. The partial distances are added to get the actual distance (reduce-add). Since p is small enough, each thread can keep the reference data in registers and thus avoid reloading the full compared data every time a different position is tested. Moreover the memory accesses are optimized as each thread loads consecutive data. Compared to Section 4.4.2, this algorithm requires much more communication between threads. However, some hardware support specific fast instructions to share data between threads, which can be used to implement an efficient reduce-add operation. For hardware with such support, this variant can be faster than the previous algorithm.

NL-means' averaging and aggregation

Design choices

Once the best neighbors and their distances are computed, patch distances need to be converted to NL-means weights. We chose to do this computation before saving the results of the patch search, thus saving directly NL-means weights. While it is possible to do both averaging and aggregation at the same time (using a scheme inspired from Section 7), we did not obtain good performance. Thus we separated the averaging and the aggregation steps. The memory layout of the proposed improved NL-means is presented in Figure 4.3.

Naive averaging

In the naive algorithm, a thread computes the averages for a single patch. In order to keep register usage low, all computations are divided per row, moving to the next row only when the results of the previous row has been saved for aggregation.

Optimized averaging

GPUs have specific instructions to access several consecutive data elements per threads. However depending on the data type and the patch width p, it is not possible to read or save a whole patch row in a single memory instruction when using a single thread. Keeping that in mind, in this optimized variant several threads are used to reduce even further the number of memory commands required. Instead of assigning one thread per patch, p threads can be assigned per patch. Each thread handles the averaging for a column of the patch. Reading or writing a row is done in one memory command, thus guaranteeing optimal memory access patterns. The resulting kernel is summarized on Algorithm 13.

Algorithm 13: Proposed NL-means filtering kernel

Input: Input image u, the output accumulator's numerator num and denominator den, the positions of the nearest neighbors positions and the corresponding weights w, the reference patch position and the column j to process 1 mean ← 0 2 var ← 0 3 for i from 0 to p-1 do Accumulate denoised_pixel on num and den with bilinear weighting

Aggregation using atomics

For NL-means, only the results of the denoised version of the reference patch are aggregated (contrarily to BM3D and VBM3D where all similar patches are denoised and aggregated). Thus, the number of patches covering a given pixel aggregated together is known in advance. In this case, it is not required to keep count of the number of covering patches for each pixel and their weights. Each result can be added with atomics on a single buffer, which will then be normalized correctly for each pixel. Atomics are necessary because several threads from different warps can write at the same positions. However, the normalization code is required to handle explicitly all patch sizes and step. Thus our code does not implement this optimization, and simply uses an accumulation buffer for the sum of the weighted denoised patches, and a second accumulation buffer for the sum of the weights. Then a simple shader divides the former by the later.

NL-means weights being floats, the averaging results are floats. Depending on the data dynamic and the required precision, int or long atomics can be used to aggregate data thresholded up a given precision (for example 1.000123 gets written as the integer 10001). Float atomics can also be used, but unfortunately to date most hardware do not support float atomics and hardware which do don't support them in OpenCL. Emulating float atomics can be done with int atomics, but with a significantly higher cost. Indeed, a float add needs to be implemented as several int atomic operations: 'reading the data', 'writing the data plus our result if the data is still set to the one we read previously' (this operation returns the value of the data before the operation). This needs to be repeated if the return value of this last operation is not what we expected (it means other threads updated the value after you accessed it). This atomic float add emulation is

Aggregation without atomics

As said in Section 19, for fixed p and step, only one accumulation buffer is enough for the aggregation since the patch aggregation weights are known in advance. If step is large enough, for example p = 8 and step = 4, it can be advantageous to not even use an accumulation buffer and thus avoid atomics completely. Instead of having a single buffer containing the aggregated value, we write the different values to aggregate per pixel to several different buffers (four buffers in the example above). These buffers can then be read, summed and normalized all a once to get the final output.

Algorithm 15: Proposed BM3D filtering kernel (for the first pass, when the maximum number of neighbors is 8, the 2D transform is the DCT and the 1D transform is the Hadamard transform)

Input: Input image u, the output accumulator's numerator num and denominator den, the positions of the nearest neighbors positions, the reference patch position to process for this kernel call

BM3D and VBM3D's filtering and aggregation

Design choices

Both BM3D's first and second pass as well as VBM3D's first pass use 8 × 8 patches. While VBM3D uses 7 × 7 patches for its second pass. We believe implementing VBM3D's second pass using 8 × 8 patches does not change the algorithm performance (see Section 4.2.3). For this reason, we will focus on using 8 × 8 patches only. The number of considered patches per location is always a power of two, 1, 2, 4 or 8 (VBM3D and BM3D), 16 or 32 (BM3D).

One way of solving the filtering is first to read the nearest neighbors for each position, form the associated 3D stack of patches, and write it to memory. The filtering can then be implemented as successive passes on the buffer containing all the stacks. However, the bandwidth needed to write and process the 3D stacks would lead to a suboptimal algorithm. Instead we propose to use the high number of registers available on a GPU to save bandwidth. All considered hardware have workgroups of 64 threads and at least 128 registers available per thread. Thus it is possible to store the 8 × 8 × 32 elements of a 3D stack of patches in the registers of the threads of a given workgroup.

Our proposal is to assign a workgroup of 64 threads for each position to process. The 3D filtering can be implemented as three separate 1D filtering operations (one for each dimension), followed by a shrinking operation (thresholding or Wiener) and finally the three separate inverse 1D filtering operations. At all times, one thread only has direct access to a subset of the 3D stack, such as a few rows. To perform the 3D filtering, each thread performs a 1D filtering operation, and then exchanges its data with the other threads so to have a new dimension of the data (for example columns instead of rows). We call that operation transposition. A transposition, for example, moves from a state where each thread contains one (or a few) patch column(s) to a state where each thread contains one (or a few) patch row(s). Implementing efficient 1D filtering or shrinking operations is not hard, thus in the following we focus on the transposition operations.

Patch search Filtering

Step 1

Step 2 The overall pseudo-code of BM3D's filtering kernel is presented in Algorithm 15 for the specific case of the first pass with a maximum number of 8 neighbors. For 16 or 32 neighbors, each thread reads not one, but two or four rows, respectively, and the operations are repeated for each. The memory layout of the proposed design choices is presented in Figure 4.4.

Efficient transpose operations

We describe here the case of eight patches (of width p = 8) and will then extend to the other number of patches. Initially, each of the 64 threads contains a column of one of the patches. Thus each thread contains eight elements and can apply one of the 1D transforms.

swapping columns and rows The first step is to swap the data so that each thread gets one row of one of the patches instead of a column. To proceed with the swap, each thread writes the eight pixel values it has in local shared memory, which is a fast memory reserved for the 64 threads. Then each thread can read from the local shared memory its target eight pixel values. Doing that efficiently is not as simple as it sounds. Extra care must be taken when reading and writing data so to achieve good performance.

Let us first formalize the transpose operation. It starts with values stored in a table T of size [8][8][8] representing the 3D patch stack. T is stored in the threads, each thread containing the section [z i ][.][y i ] (corresponding to the column y i of patch z i ) where i is the thread index (from 0 to 63), and z i , y i thread coordinates, with y i = i mod 8 and z i = i/8 . The goal is to have the data reorganized as in a second table T such as

T [z i ][y i ][.] = T [z i ][.][y i ] .
In practice the memory of T is reused for T .

The most "naive" way of performing the swap is to simply write T in local shared memory in an arbitrary order, and then read the indices in the right order. Examples of such "naive" algorithms are described in Algorithms [START_REF] Wzs + | [END_REF]

T [64y i + 8z i + k] ← T [k] 3 barrier()
// Wait for all threads to finish writing 4 for k from 0 to 7 do 5

T [k] ← T [64k + 8z i + y i ]
We shall analyze more in depth Algorithm 17. In this naive version, the data is written into an intermediate 3D

i ][z i ][.] = T [z i ][.][y i ] . It just remains to read T in the right order to obtain T , T [z i ][y i ][.] = T [.][z i ][y i ] .
Elements are written and read one by one in each thread (loop on k in Algorithm 17), thus resulting into eight write instructions and eight read instructions for each thread (ignoring the instruction merging mentioned later). While at read time, the threads load 64 consecutive elements and thus guaranteeing no bank conflicts, at writing time the proposed pattern triggers bank conflicts Indeed threads 0, 4, 8, etc, write in their first call at cases 0, 32, 64, which map to the same memory bank (all current hardware have either 16 or 32 banks). The same problem happens with the other similar groups of threads. The bank conflicts cause the accesses to the same banks to be serialized, thus being executed as several separate calls. The write performance will thus be suboptimal. The solution to improve performance is to 'shift' the locations of the data written by the threads. The corrected algorithm is identical except T has padding. It is of size

[8][64 + x] , but is interpreted as a table of size [8][8][8]
as before, but with an offset of x times the index of the first dimension.

This performance trick is well documented and is described for example in [START_REF]NVIDIA OpenCL Best Practices Guide[END_REF], which suggests x = 1 in the case of a matrix transpose. However, it does not give the best performance on all hardware: Indeed optimal performance is achieved by maximizing the use of banks per calls. However, some hardware can write or read several consecutive elements per thread in one call. The tested INTEL hardware, which can have a warp size of eight, can support reading or writing four elements per thread in one call. Reading or writing only one element results in suboptimal performance as the banks are underutilized. The tested AMD hardware, which has a warp size of 64, executes local shared memory commands in four parts, first the first 16 threads, etc, until the last 16 threads, and ideally each of these subcalls should use all the banks (they have 32). Like for the INTEL GPU, the tested AMD GPU can write or read several consecutive elements per thread in one call (two per call). The tested NVIDIA hardware has warps of 32 threads, has 32 banks, and does not process them in two subcalls. Note that we assume that a bank covers elements of four bytes, which is the size of a float (The NVIDIA hardware can configure it to eight bytes).

Based on this variety of behaviors, it is difficult to design read or write patterns that are optimal on all hardware as it depends on the warp size, the number of banks and how they are accessed. However, assuming the number of banks is a divisor of 64 -our workgroup size -, we noticed that having each thread write (or read) the maximum number of consecutive elements it can write (or read) in one instruction is always an optimal pattern. The same can be noticed when having threads write (or read) from consecutive addresses modulo 64 (thus to consecutive banks). We will use these access patterns in our algorithms for best performance and compatibility (we expect these patterns will still be optimal for future generations).

This leads us to Algorithm 18. The best x corresponds to the maximum number of consecutive elements the hardware can read or write per thread per access. Thus, for the tested hardware, x =

i + 8z i + k] ← T [k] 3 barrier()
// Wait for all threads to finish writing 4 for k from 0 to 7 do 5

T [k] ← T [(64 + x)k + 8z i + y i ]
4 should be best for INTEL, x = 2 for AMD, and x = 1 for NVIDIA. We verified experimentally that indeed these parameters proved to be the best performing for each hardware. We also checked the generated assembly code to confirm that the compiler combines the writes of consecutive data into write packets of x elements. We expect the best value of x to change for future hardware generations and should be set accordingly. When memory organization and the memory command packing abilities are not known for a specific hardware, the best x can be determined empirically.

The proposed writing order covers all the banks per groups of threads in a warp and uses all the banks. It is therefore the best possible pattern. For example, on the INTEL hardware with x = 4, the first write of thread 0 writes to banks 0, 1, 2, 3, thread 1 writes to banks 4, 5, 6, 7, etc. This corresponds indeed to the optimal pattern we suggested.

Swapping rows and channels Thanks to the transpose presented previously, the first two 1D transforms can be applied on both the rows and the columns. The last 1D transform must be applied on the third dimension of the 3D patch stack. This means that each thread must contain the third dimension of the 3D stack for a given patch pixel. The only constraint for this transposition is for the position (0, 0) of each transformed patch. The rest of the elements does not require any specific ordering as long as the inverse transposition puts back the data in the correct thread. In our case we will require that the position (0, 0) of each transformed patch is put in the thread 0. Furthermore we will assume x is either 1, 2, 4 or 8.

To

T [z i ][j/x + z i * 8][y i ][j mod x] = T [z i ][y i ][j].
This writing pattern verifies the conditions set in the previous paragraph. It avoids bank conflicts and enables full use of the ability of the hardware to write x elements in one call. At reading time, we assign

T [j][z i ][y i ] = T [j][a i + j * 8][b i ][c i ] , where a i * 8 * x + b i * x + c i = z i * 8 + y i .
As among threads, y i increases first, then z i , the pattern also prevents bank conflicts (it verifies again the conditions set previously). Contrary to when writing T , the read commands cannot be combined to read more than one value per call.

One problem with this proposed algorithm is the size of T which can be huge: It is of length 512(1 + x). However T can be fit in a much smaller table of size

[8][8/x][8][x]
, that is of length 512, while maintaining an optimal access pattern. To do so, modulus will be used in the writing and read address computations.

Algorithm 19 presents the entire algorithm (with T a 1D table of length 512). It is easier to visualize on Algorithm 19 that the algorithm is correct and that it indeed gives the pattern we described in the previous paragraph: Since (8z i + y i ) indexes threads from 0 to 63, the loops of Algorithm 19: Swapping rows and channels Input: z i , y i thread indices, T table of 8 registers for the thread, T table in local shared memory 1 for h from 0 to (8/x -1) do 2 for j from 0 to x-1 // The x memory operations are merged

3 do 4 T [64z i + (8z i x + 8hx + y i x + j) mod 64] ← T [hx + j]
5 barrier() // Wait for all threads to finish writing 6 for k from 0 to 7 do 7

T [k] ← T [64k + (8kx + 8z i + y i ) mod 64]
Algorithm 19 access indeed consecutive addresses modulo 64 for a fixed h and a fixed k. Thus optimal performance is guaranteed.

Generalizing to 1, 2, 4, 16 or 32 patches The swapping operations presented previously assumed we have to process eight patches. In order to generalize to more patches, we simply need to store several columns of data initially per thread, and call the transposition functions presented previously several times. When having fewer patches, some operations could be avoided. However, we simply called the function for eight patches while setting the missing patches to 0 and therefore just ignoring them in the 1D transforms.

On the use of atomics for the aggregation

Due to the non-uniform weighting, the weighted pixel values being aggregated have a wide range and, contrarily to NL-means, many elements are aggregated per pixel for BM3D and VBM3D. We found that even with 8-bit RGB images, 32 bits integers could be insufficient to store the weighted sum in some corner cases. Long atomics (64-bits integers) are recommended. When not supported, the emulation of float atomics discussed in the Section 19 should be used. Integer atomics could be used if reducing the range of the weights used for the aggregation, though the PSNR could be affected.

Conclusion

In this section, we have explained in detail different optimized implementations (as well as their naive versions) of the pieces of code required for NL-means, BM3D and VBM3D. In our implementation, which we compare to other implementations in Section 4.5, we have selected the best optimized codes among the ones described, depending on the parameters. For the patch search of NL-means and BM3D, we use the fast variant described in Section 7 when possible (p = 8, step = 4), else use the variant of Section 7, except for step = 1, in which case the variant of Section 7 is fastest. The naive variant is only used for border handling when needed. For NL-means, we use the optimized averaging, rather than the naive one. We do aggregation using atomics for all methods. Last, VBM3D's patch search uses Section 4.4.2 for INTEL GPUs, else the algorithm of Section 4.4.2.

Benchmarks

In this section we will compare the performance, both in terms of running time and PSNR, of our implementations with several reference implementations. To remove the effects of differences in color handling, all the experiments in this chapter are grayscale. The denoising performance will be compared on several datasets: BSD68 a set of 68 images from the Berkeley segmentation dataset [START_REF] Martin | A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[END_REF], Kodak a set of 24 images from the Kodak dataset [START_REF] Franzen | Kodak lossless true color image suite[END_REF] and IPOL a set of 16 images from [START_REF] Colom | Multiscale noise estimation and removal for digital images[END_REF] 1 . In addition, we compare the denoising performance and running time for two images: Lena and CMLA (from [START_REF] Briand | Optimization of Image B-spline Interpolation for GPU Architectures[END_REF]). The former is a 512 × 512 image, while the latter is 4608 × 3456. Table 4.3 shows the different OpenCL devices, called accelerators (acc.) later on, considered. We consider a high-end multicore CPU (0), an integrated GPU (1), and middle-range consumer grade GPU (2) and a high-end GPU (3).

NL-means

In this section, we compare our GPU implementation to the reference CPU code of [START_REF] Buades | Non-local means denoising[END_REF] and to OpenCV (version 4.1.0 [START_REF] Bradski | The OpenCV Library[END_REF]) with both a CPU and a GPU implementation.

We compare the denoising performance with additive white Gaussian noise of standard deviation σ = 20. For this noise level, [START_REF] Buades | Non-local means denoising[END_REF] recommends 5 × 5 patches and h = 0.4σ = 8. However, likely due to differences in the implementation choices, h = 8 did not give good results for OpenCV. By maximizing the PSNR on a serie of images of the Waterloo dataset [MDW + 17], we found the best parameter for OpenCV to be h = 20 for this noise level with 5 × 5 patches. Due to the differences in our implementation (number of neighbors, patch size, flat patch trick, aggregation), a different parameter h was needed as well. We used h = σ. We compare our implementation for two patch sizes. By default, our implementation uses 8 × 8 patches with a subgrid step of four. Thus a pixel is covered by four patches. We also show results for patches of size 5 × 5 and a subgrid step of one, which corresponds to the parameters of the reference implementation.

We illustrate the denoising performance of the implementations on standard datasets in Ta- ble 4.4. As can be seen, OpenCV performs about the same with the CPU and GPU backends, but underperforms compared to the reference implementation. Our code with the default parameters (Ours) was slightly outperformed by the reference implementation on BSD68 and Kodak, but performed better on the IPOL dataset. When analyzing the results, one can see that our method has better denoising performance on images with flat regions thanks to the flat patch trick. Due to the use of larger patches than recommended for this noise level, the performance is slightly lower elsewhere. When using our code with 5 × 5 patches and a subgrid step of 1 (Ours P5, S1), our method outperforms the reference implementation on all datasets tested. The gains over [START_REF] Buades | Non-local means denoising[END_REF] can be explained by the use of a bilinear scheme for the combination of the results of overlapping patches, instead of using a constant weight, and by the flat patch trick. In Table 4.5, we compare both the denoising performance (PSNR) and the execution time on Lena and CMLA. One can observe that OpenCV's CPU implementation is more than 30 times faster than the reference code, although at the cost of a lower PSNR. Our GPU code runs significantly faster with the default parameters than with 5 × 5 patches and a subgrid step of 1, but the denoising quality is slightly lower. Both compared versions of our code ran faster than OpenCV's GPU implementation on Lena, while obtaining better denoising results. OpenCV could not denoise CMLA on the GPU as it required intermediate buffers that would not fit in the 12GB GPU memory. One can see that the execution time of our implementation does not scale linearly with the image resolution. The difference in running time per pixel between both images can be explained by the effects of cache (Lena fits completely in the GPU cache) and better use of the parallelization capabilities of a GPU in the case of CMLA which is bigger. The running time of our implementation on three different accelerator is analyzed in of the results (normalization of an aggregation buffer) is small and therefore not displayed. It is nonetheless included in the total running time. Accelerator 1 could not compute the result on CMLA as the execution time was more than the timeout threshold of this GPU. We can notice our code executing on CPU (accelerator 0) is competitive with the compared CPU implementations (the equivalent single core time is 31s, compared to OpenCV's 16s and [BCM11]'s 522s), even though our code was not optimized for CPU specifically. This is likely due to an efficient use of the CPU extended instruction set (SSE, AVX, etc) for the code generated by OpenCL.

BM3D There are many different implementations of BM3D available. We compare our code to two GPU implementations, [HK19, HK18] and [START_REF] Wang | Gpu-accelerated denoising with bm3d[END_REF], and two CPU implementations, [START_REF] Lebrun | An Analysis and Implementation of the BM3D Image Denoising Method[END_REF] and OpenCV's. [START_REF] Lebrun | An Analysis and Implementation of the BM3D Image Denoising Method[END_REF] is a reference CPU re-implementation of the original paper and OpenCV is the implementation that can be found in OpenCV. We only consider open-source implementations reproducing the original method described in [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF].

In Table 4.7, we show the default parameters of these methods as well as which parameters can be modified. No other methods use the default parameter proposed in the original BM3D publication. Only our method can be configured to have exactly these parameters. It has to be noted, though, that the Hadamard and Haar transforms have been shown to give equivalent results in [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF], thus arguably, [START_REF] Lebrun | An Analysis and Implementation of the BM3D Image Denoising Method[END_REF] comparable. Thus, we only compare using the same parameters, [START_REF] Lebrun | An Analysis and Implementation of the BM3D Image Denoising Method[END_REF], [START_REF] Honzátko | Accelerating block-matching and 3d filtering method for image denoising on gpus[END_REF] and our implementation. The patch search region is set to 39 × 39, the 2D transforms are both set to DCT, and the 1D transforms to Hadamard. The patch step is 3, and the maximum number of nearest neighbors is 16 for the first pass, and 32 for the second pass. We also compare the performance of all implementations with their default parameters. Similarly as before, we show the denoising performance on standard datasets in Table 4.8 and denoising performance and running time for two images in Table 4.9. Details about the running time of our method can be found in Table 4.10. As we can see, OpenCV and [WS17] underperform compared to other methods. This is likely due to their choice of 2D or 1D transforms.

The three compared methods set to the same fixed parameters had similar denoising performance, which validates the equivalence of these implementations, besides the default parameters. However our implementation was faster than the other GPU implementation [START_REF] Honzátko | Accelerating block-matching and 3d filtering method for image denoising on gpus[END_REF]. The main optimization compared to [START_REF] Honzátko | Accelerating block-matching and 3d filtering method for image denoising on gpus[END_REF] is the use of a single kernel to implement all the filtering operations (see Section 4.4.4). When comparing the performance of all the implementations with their default parameters, [START_REF] Lebrun | An Analysis and Implementation of the BM3D Image Denoising Method[END_REF] had the best denoised results, and our implementation was the fastest. Our default parameters were chosen to maximize denoising performance while looking for aggressive speedups. As a result, it ran almost nine times faster than the closest competitor. However as our parameters can easily be changed, our code can also be used to reproduce the results of the best performing [START_REF] Lebrun | An Analysis and Implementation of the BM3D Image Denoising Method[END_REF]. This is done at a cost similar to the one reported in Table 4.9 as the main difference between the parameters suggested in [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF] and the fixed one used for the table is the use of Bior instead of DCT for the first pass. Table 4.10, which analyzes the running time of our method on several accelerators, shows that, contrary to NL-means, most of the running time is spent in the filtering of the patch stacks, rather than in the search of neighbors. One limiting factor of this filtering is the writing of the results on an accumulation buffer with emulated float atomics. As said in Section 7, hardware long atomics can be used if the range of the input is known. We can see in Table 4.10 that accelerator 2 benefits significantly from hardware long atomics since the total running time is almost divided by two. The hardware of accelerator 3 is able to do hardware float atomics and long atomics (available in CUDA), but these functionalities are not available for the OpenCL driver at the time of writing. Both implementations [START_REF] Honzátko | Accelerating block-matching and 3d filtering method for image denoising on gpus[END_REF] and [START_REF] Wang | Gpu-accelerated denoising with bm3d[END_REF], written in CUDA, used hardware float atomics. We notice that, like it was for NL-means, our BM3D code running on accelerator 0 (CPU) is competitive with the compared CPU implementations. It runs at an equivalent singlecore execution time of 159s, compared to 205s for OpenCV and 425s for [START_REF] Lebrun | An Analysis and Implementation of the BM3D Image Denoising Method[END_REF]. One has to be careful though when comparing these results because the parameters are different.

VBM3D and video NL-means

In this section, we delve into the performance of the video variations of the previous algorithms.

While NL-means has been previously used for video denoising, there is no standard video version. Thus, for this comparison, we use our code extending the search to a 3D window including the 4 past images and the 4 future images of the current sequence (similarly to VBM3D). In addition, we use 16 × 16 patches. The use of bigger patches is justified for video denoising as it results in better matches of the same content in the previous frames, which improves the denoising result and the temporal consistency.

We compared with VBM3D using the original parameters of [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF] as implemented by [START_REF] Ehret | Implementation of the vbm3d video denoising method and some variants[END_REF], and our implementation (which has a different patch size for the second pass, as explained in 4.2.3). To highlight the improvements of the video algorithms over the image versions, we show the denoising performance of NL-means and BM3D with our default parameters, used separately on each frame (no 3D search window). dataset of the compared implementations. The Derf dataset corresponds to the Derf's Test Media collection2 as processed in [START_REF] Arias | A comparison of patch-based models in video denoising[END_REF]: The original videos are RGB of size 1920 × 1080, and seven grayscale sequences of 100 frames were extracted and down-sampled by a factor two (the resolution is thus 960 × 540).

Video methods obtain a significantly superior PSNR (a 1.99 db gain for NL-means and 1.28db for BM3D). While NL-means is slower in its video version than its image version, due to the bigger patch search region, VBM3D is faster than our BM3D with default parameters. This is due to the competitive video patch search algorithm of VBM3D, and due to the first pass of the algorithm using a patch step of 6 for VBM3D, while our BM3D uses 4. The fact that our GPU implementation has a slightly better denoising performance (of 0.10 db) compared to the reference CPU implementation is due to the use of 8 × 8 patches in the second pass. This phenomenon was already observed in Table 4.2, on the same dataset (with a different noise generation). For VBM3D, Table 4.12 shows that computation time is quite balanced between all steps even though the search takes slightly more time than the filtering. It's also important to notice that the implementation achieves real-time denoising on these sequences for both accelerators 2 and 3.

Conclusion

In this chapter, we proposed a GPU implementation of NL-means, BM3D and VBM3D. The technical challenges of this endeavor and of the proposed solutions have been discussed. The PSNR denoising performance was shown to match, and sometimes even to surpass thanks to careful implementation choices, the denoising performance of the reference CPU implementations while being several orders of magnitude faster. We also proposed compromises for the default parame-ters to get an even larger speedup at a minor cost for the denoising performance. This work paves the way to new uses of these algorithms in time constrained environments.

5 Non-Local video denoising by CNN

Non-local patch based methods were until recently the state of the art for image denoising but are now outperformed by CNNs. In video denoising however, they are still competitive with CNNs, as they can effectively exploit the video temporal redundancy, which is a key factor to attain high denoising performance. The problem is that CNN architectures are not compatible with the search for self-similarities. In this chapter we propose a simple, yet efficient way to feed video self-similarities to a CNN. The non-locality is incorporated into the network via a first non-trainable layer which finds for each patch in the input image its most similar patches in a search region. The central values of these patches are then gathered in a feature vector which is assigned to each image pixel. This information is presented to a CNN which is trained to predict the clean image. We apply the proposed method to image and video denoising. In the case of video, the patches are searched for in a 3D spatio-temporal volume. The proposed method achieves state-of-the-art performance. This work has been published in [DEM + 19] and [DEM + 20].

Introduction

Recently, we have seen denoising methods moving away from patch-based methods presented in Part I and a new trend using machine learning for denoising emerging.

Image denoising has a vast literature where a variety of methods have been applied: PDEs and variational methods (including Markov random fields, MRF, models) [ROF92a,CCC + 10,Rot05], transform domain methods [START_REF] Donoho | Ideal spatial adaptation by wavelet shrinkage[END_REF], non-local (or patch-based) methods [START_REF] Buades | Denoising image sequences does not require motion estimation[END_REF][START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF], multiscale approaches [START_REF] Facciolo | Conservative scale recomposition for multiscale denoising (the devil is in the high frequency detail)[END_REF], etc. See [START_REF] Lebrun | Secrets of image denoising cuisine[END_REF] for a review. In the last two or three years, CNNs have taken over the state of the art. In addition to attaining better results, CNNs are amenable to efficient parallelization on GPUs potentially enabling real-time performance. We can distinguish two types of CNN approaches to image denoising: trainable inference networks and black box networks.

In the first type, the architecture mimics the operations performed by a few iterations of optimization algorithms used for MAP inference with MRFs prior models. Some approaches are based on the Fields-of-Experts model [START_REF] Roth | Fields of experts: a framework for learning image priors[END_REF], such as [Bar09, SR14, CP17]. The architecture of [START_REF] Vemulapalli | Deep gaussian conditional random field network: A model-based deep network for discriminative denoising[END_REF] is based on EPLL [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF], which models the a priori distribution of image patches as a Gaussian mixture model. Trainable inference networks reflect the operations of an optimization algorithm, which leads in some cases to unusual architectures, and to some restrictions in the network design. For example, in the trainable nonlinear reaction diffusion network (TNRD) of [START_REF] Chen | Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration[END_REF] even layers must be an image (i.e. have only one feature). As pointed out in [START_REF] Kobler | Variational networks: Connecting variational methods and deep learning[END_REF] these architectures have strong similarities with the residual networks of [START_REF] He | Deep residual learning for image recognition[END_REF].

The black-box approaches treat denoising as a standard regression problem, not using much of the domain knowledge acquired during decades of research in denoising. The first denoising approaches using neural networks were proposed in the mid and late 2000s. Jain and Seung [START_REF] Jain | Natural image denoising with convolutional networks[END_REF] proposed a five layer CNN with 5 × 5 filters, with 24 features in the hidden layers and sigmoid activation functions. Burger et al. [START_REF] Burger | Image denoising: Can plain neural networks compete with bm3d?[END_REF] reported the first results competitive with patch-based methods using a multilayer perceptron trained to denoise In [START_REF] Cruz | Nonlocality-reinforced convolutional neural networks for image denoising[END_REF] the authors propose an iterative approach that can be used to reinforce non-locality to any denoiser. Each iteration consists of the application of the denoiser followed by a non-local filtering step. An inconvenience is that the resulting algorithm requires to iterate the denoising network. Trainable non-local modules have been recently proposed by using differentiable relaxations of the nearest neighbor [LWF + 18] and k nearest neighbors [START_REF] Plötz | Neural nearest neighbors networks[END_REF] selection rules, or using Graph CNNs [START_REF] Valsesia | Image denoising with graph-convolutional neural networks[END_REF], where the graph edges encode the non-local connections between pixels with similar features. These approaches are very interesting as they allow the combination of local and non-local interactions in a trainable way.

CNNs have been successfully applied to several video processing tasks such as deblur-ring [SDW + 17], video frame synthesis [LYT + 17] or super-resolution [HWW15, SVB18], but their application to video denoising has been limited so far. In [START_REF] Chen | Deep rnns for video denoising[END_REF] a recurrent architecture is proposed, but the results are below the state of the art. More recently, Tassano et al. [START_REF] Tassano | Dvdnet: A fast network for deep video denoising[END_REF] proposed DVDnet, a convolutional architecture which processes five consecutive frames to predict the central frame. Each frame is first denoised spatially, and then warped to frame t via an optical flow. The aligned frames are stacked together with the central frame and processed by a "temporal denoising" network. The authors use a non-trainable optical flow, which prevents the network from being trained end-to-end. Two recent works proposed networks without explicit motion estimation: ViDeNN-G [CvG19] processes three consecutive frames, and applies first a spatial denoising followed by temporal denoising, similar to [START_REF] Tassano | Dvdnet: A fast network for deep video denoising[END_REF], except that the frames are stacked without aligning them. A different architecture, named fastDVDnet, was proposed in [START_REF] Tassano | Fastdvdnet: Towards real-time video denoising without explicit motion estimation[END_REF]. Instead of first using a spatial denoising, three consecutive noisy frames are stacked together. The stack is processed by a U-net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] which predicts the central frame.

To extend the temporal receptive field of the network, the authors cascade two levels of these networks. The overall network takes five frames as input. Recently [GMU18b, MBC + 18a, EDAF19] focused on the related problem of image burst denoising reporting very good results. There is also recent work focusing on unknown noise-model denoising for videos that use self-supervision for training [EDM + 19, EDAF19].

Before the advent of CNNs, patch-based methods were the state of the art [DFE07, MBFE12, AM18b, EAM17, BLM16, WLPB17], and some continue to be competitive in terms of denoising performance [AM18b,EAM17], albeit with a larger computational cost). They exploit extensively the self-similarity of natural images and videos, namely the fact that most patches have several similar patches around them (spatially and temporally). Each patch is denoised using these similar patches, which are searched for in a region around it. The search region generally is a space-time cube, but more sophisticated search strategies have also been used. Because of the use of such broad search neighborhoods these methods are called non-local. While these video denoising algorithms perform very well, they often are computationally costly.

Patch-based methods usually follow three steps that can be iterated: (1) search for similar patches, (2) denoise the group of similar patches, (3) aggregate the denoised patches to form the denoised frame. VBM3D [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF] improves the image denoising algorithm BM3D [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF] by searching for similar patches in neighboring frames using a "predictive search" strategy which speeds up the search and gives some temporal consistency. VBM4D [START_REF] Maggioni | Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms[END_REF] generalizes this idea to 3D patches. In VNLB [START_REF] Arias | Towards a bayesian video denoising method[END_REF], video extension of [START_REF] Lebrun | A nonlocal bayesian image denoising algorithm[END_REF], spatio-temporal patches that were not motion compensated are used to improve the temporal consistency. In [START_REF] Ehret | Global patch search boosts video denoising[END_REF] a generic search method extends every patch-based denoising algorithm into a global video denoising algorithm by extending the patch search to the entire video. SPTWO [START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF] and DDVD [START_REF] Buades | Dual domain video denoising with optical flow estimation[END_REF] use optical flow to warp the neighboring frames to each target frame. Each patch of the target frame is then denoised using the similar patches in this volume with a Bayesian strategy similar to [START_REF] Lebrun | A nonlocal bayesian image denoising algorithm[END_REF]. Recently, [START_REF] Wen | Joint adaptive sparsity and low-rankness on the fly: an online tensor reconstruction scheme for video denoising[END_REF] proposed to learn an adaptive optimal transform using batches of frames.

Patch-based approaches have also been applied in frame-recursive denoising methods [START_REF] Ehret | Non-local kalman: A recursive video denoising algorithm[END_REF][START_REF] Arias | Kalman Filtering of Patches for Frame-Recursive Video Denoising[END_REF], that denoise each frame using only the corresponding noisy frame and the previous denoised frame.

It works particularly well in the case of video denoising, where it achieves state-of-the-art performance.

The method first computes for each image patch the n most similar neighbors in a rectangular spatio-temporal search window and gathers their central pixels forming a feature vector which is assigned to each image location. This results in an image with n channels, which is fed to a CNN trained to predict the clean image from this high dimensional vector. We trained our network for grayscale and color video denoising. Practically, training this architecture is made possible by a GPU implementation of the patch search that allows computing the nearest neighbors efficiently.

To train our network we created a dataset of 17k video segments. In the two testing datasets, our network obtains state-of-the-art results on both color and grayscale video denoising. The code to generate the datasets and reproduce our results is available online1 . A preliminary version of this work was presented in [DEM + 19]. The present version includes an extension to color videos, a detailed comparison with recent works, extended discussions comparing these methods, and new experiments.

Proposed method

Let u be a grayscale video and u(x, t) denote its value at pixel position x in frame t. We observe v, a noisy version of u contaminated by additive white Gaussian noise: v = u + r, where, r(x, t) ∼ N (0, σ 2 ).

(5.1)

Our video denoising network processes the video frame by frame. Before it is fed to the network, each frame is pre-processed by a non-local patch search module which computes a non-local feature vector at each image position. A diagram of the proposed method is shown in Figure 5.1. We call our method VNLnet for Video Non-Local Network.

Non-local features

Each frame in the video is pre-processed by the non-local patch search module to produce a 3D tensor f nl of n channels. This is the input to the network. The parameter n is the number of nearest neighbor patches searched for each pixel by this pre-processing module.

Let P be a patch of size s × s centered at pixel x in frame t. The patches are arranged as vectors with s 2 components. The patch search module computes the L 2 distances between the patch P and the patches in a 3D rectangular search region of size w s × w s × w t centered at (x, t). The positions of the n most similar patches are (x i , t i ), ordered either by increasing distance or by increasing frame index t i .

The pixel values at those positions are gathered to produce the n-dimensional non-local feature vector associated to pixel (x, t):

f nl (x, t) = [v(x 1 , t 1 ), ..., v(x n , t n )].
(5.2)

The non-local features correspond to n images which resemble the frame v(•, t) (see Figure 5.2). One of them is v(•, t) itself, as the reference patch P is always among the n nearest neighbors. The remaining n -1 images are built from the central pixels of the most similar patches to each patch in v(•, t).

One neighbor per frame. We will also consider a restricted version of the patch search, not allowing more than one match per-frame. In this variant, we set the number of matches n to be equal to the number of frames in the search region, resulting in a match for each frame. The neighbors are sorted by frame index instead of the patch distance. With this configuration, the ith non-local feature map corresponds to a warped version of the ith neighboring frame, aligned to the reference frame t. An example is shown in Figure 5.2, for n = 15. This can be related to [TDV19a, XCW + 19], which align the input frames using an optical flow. Indeed patch matching can be seen as a rough optical flow with integer displacements.

Other alternatives. The proposed non-local features are inspired by classical patch-based methods such as [START_REF] Buades | A non-local algorithm for image denoising[END_REF]. Selecting similar patches with the L 2 distance and extracting their central pixel is possibly the simplest way of providing the network with a non-local context.

The L 2 distance is widely used, and was shown to be optimal for additive white Gaussian noise (AWGN) [START_REF] Deledalle | How to compare noisy patches? Patch similarity beyond Gaussian noise[END_REF]. The same work also shows that other patch distances should be used for other noise distributions. We considered some alternatives to feeding only the central pixels to the network: such as providing as well the patch similarities (which could be used by the network to ignore bad matches), or extracting a small patch (3 × 3 or 5 × 5) around the center of every matching patch, instead of the central pixel. However, we did not observe a significant improvement to justify the increase in complexity: the differences in the validation PSNR for these variants were within a 0.1dB range. The reason for this is probably that the added pixels in the patch around the central pixel are redundant with the central pixels of the similar patches for neighboring pixels.

In a previous version of our work [DEM + 19] we also included four 1 × 1 convolutional layers as a trainable transformation of the non-local features. We removed them in this work as we found that the same performance can be achieved with a simpler design.

Network architecture

For processing the non-local features, we considered standard architectures used in image restoration modifying their input layer to the n channel input tensor. We have tested three such architectures.

DnCNN. The DnCNN architecture was proposed in [ZZC + 17b] for still image denoising. It consists of a feed-forward network with 17 layers with 64 3 × 3 convolution kernels, each one followed by batch normalization and ReLU activations. The output layer is a 3 × 3 convolution. The network outputs a residual image, which has to be subtracted to the noisy image to get the denoised one.

U-Net. U-Nets (also called hourglass networks) have been introduced in [RFB15] for image segmentation. This architecture consists of an encoder-decoder convolutional network. The encoder part downscales the input by a series of pooling layers or strided convolutions, whereas the decoder upscales the coarse features to the desired output resolution. The differentiating aspect of U-Nets from previous encoder-decoder networks are skip connections that bypass each coarser scale. These networks have been applied to many tasks, among them denoising of both single images [CCXK18, SMD17] and videos [START_REF] Tassano | Dvdnet: A fast network for deep video denoising[END_REF]. In our experiments, we use the U-Nets blocks of [START_REF] Tassano | Fastdvdnet: Towards real-time video denoising without explicit motion estimation[END_REF], except that the first two layers, which feature group convolutions and a number of kernels computed from the number of input frames, are replaced by two standard convolution layers of 32 kernels. + 17] is a residual CNN architecture tuned for super-resolution. The main architectural differences with DnCNN are the introduction of skip connections every two convolution layers, and the removal of the batchnorm layers. Thus in our experiments we took our DnCNN networks and applied these two changes (the number of convolutional layers and their parameters are kept).

EDSR. EDSR [LSK

Datasets and training

We denote by F the application of the network. The input to F at time t is f nl t = f nl (•, t), the nchannel image with non-local features gathered from a window of frames around t. The training loss is the mean square error (MSE) between the reconstructed frame and the ground truth clean frame:

l(F(f nl t ), u t ) = ||F(f nl t ) -u t || 2 2 .
(5.3)

We use residual training, as in [ZZC + 17b]. This means that the network actually predicts the noise, and therefore the denoised image is obtained by subtracting the predicted noise from the noisy input.

For RGB videos, we compute the patch search on grayscale frame resulting from averaging the color channels. To form the non-local features we take the RGB components of the central pixels of the matching patches, resulting in a input tensor with 3n channels. For RGB videos we only trained a DnCNN network, with 25 layers and 96 channels (i.e. about three times the number of parameters used for grayscale video).

Datasets

For the training and validation sets we used a database of short segments of 16 frames extracted from YouTube videos. Only HD videos with Creative Commons license were used. From each video we extracted several segments, separated by at least 10s. In total the database consists of 16950 segments extracted from 1068 videos, organized in 64 categories (such as antelope, cars, factory, etc.). As the original videos might contain compression artifacts, noise, etc, we downscaled the video to a height of 540 pixels. This removes the minor artifacts of the videos and better represents clean targets. In addition, we randomized the anti-aliasing filter width (Gaussian blur) of the downscaling. This results in a variety of sharpness/blur in the training dataset, and thus helps reducing dataset bias. We separated 6% of the videos of the database for the validation (one video for each category). For grayscale networks, grayscale data is obtained by converting the previous color datasets.

For training we ignored the first and last frames of each segment for which the 3D patch search window did not fit in the video. During validation we only considered the central frame of each sequence. The resulting validation score is thus computed on 503 sequences (1 frame each).

For testing, we used two distinct datasets. The first one is the dataset of [START_REF] Arias | A comparison of patchbased models in video denoising[END_REF], which was extracted from the Derf's Test Media collection. 2 It is composed of seven sequences of 100 frames of size 960 × 540. In this dataset, the camera is either still or has a smooth motion. The second one is the test-dev split of the DAVIS video segmentation challenge [PTPC + 17]. It consists of 30 videos having between 25 and 90 frames. In this dataset, the motion is more challenging. In order to remove compression artifacts and noise present in the original images, both datasets were obtained with a similar downscaling as for the training set (the original images ranged between HD and 4K). Each dataset was processed using a different anti-aliasing filter width.

Training details

At each training epoch, first a subset of the videos of the dataset is selected and noise is added to generate noisy samples. Second the non-local patch search module is run on every video selected. This results in videos of non-local features where each frame has n channels containing the output of the patch search module. Third the network is trained on mini-batches built from small crops extracted at random positions on the videos of non-local features.

During training, we ignore spatio-temporal border effects by excluding the first and last w t /2 frames and ignoring crops at borders. At testing time, we simply extended the video by mirroring it at the start and the end of the sequence and adding black borders for the patch-search module.

The training epochs comprise 17000 mini-batches of size 64 square crops of 44 pixels width. We trained for 30 epochs using a combination of recent new optimization techniques which give small performance improvements on various machine learning tasks: Lookahead [START_REF] Zhang | Lookahead optimizer: k steps forward, 1 step back[END_REF], RAdam [LJH + 19] and Gradient Centralization [START_REF] Yong | Gradient centralization: A new optimization technique for deep neural networks[END_REF]. 3 In addition, we reduced the learning rate at epochs 15 and 27 (from 1e -3 to 1e -4 and 1e -6 respectively). Training a network takes about 14 hours on a NVIDIA TITAN V for grayscale videos, and 24 hours for color videos.

Experiments and parameter tuning

In this section we evaluate the effect of the non-local features first in still image denoising, and then, after studying the impact of the parameters, in video denoising. Unless otherwise stated, the results reported were obtained using a DnCNN architecture for the fusion network. The network is trained from scratch for each different parameter setting.

The potential of non-locality

Although the focus of this work is in video denoising, it is still interesting to study the performance of the proposed non-local CNN on a single image. In contrast, as we will see below, the temporal redundancy of videos allows using very large patches.

Parameter tuning for video denoising

The non-local search has three main parameters: The patch size, the number of retained matches and the size of in the search region (both spatially and in number of frames). We expect the best matches to be past or future versions of the current patch, so we set the number of matches as the number of frames on which we search. In the following we study the impact of the different parameters. Note that for each different parameter, we retrain the fusion network to make sure that it is the optimal one for the chosen parameters. In all cases, we consider denoising of grayscale videos with σ = 20.

Patch size. In Table 5.1a, we explore the impact of the patch size used for the matching. For each patch size we show results with the unconstrained patch search and with the restriction of one neighbor per frame (ONPF). In both cases, the results improve with the patch size, even for patches much larger than the ones typically used in patch-based methods. The ONPF restriction produces a slight improvement in performance, particularly for larger patches. Figure 5.4 shows visual results obtained with the ONPF restriction. As the patch size grows, there is a noticeable increase in the amount of details recovered from the background.

Number of frames in the search region. In Table 5.1b and Figure 5.5, we see the impact of the number of frames used in the search window (and thus the number of nearest neighbors). One can see that the more frames, the better. Increasing the number of frames beyond 15 (7 past, current, and 7 future) does not justify the small increase of performance. Foreground moving objects are unlikely to get good neighbors for the selected patch size, unlike background objects, thus it comes to no surprise that the visual quality of the background improves with the number of patches, while foreground moving objects (for example the legs in Figure 5.5) do not improve much.

Spatial size of the search region. Results varying the size of the search region are shown in Table 5.1c. The results improve for larger search regions, although with diminishing returns. From 41×41 to 51×51 there is no significant improvement. Our search region is not compensated by motion, thus having a large search region is necessary to find matching patches on distant frames for moving objects.

Choice of the fusion network. We have evaluated the performance obtained with the two other architectures for the fusion network described in Section 5.2. The results obtained are shown in Table 5.2. The three networks were trained with the same training set and the same procedure. All networks achieve a very similar PSNR, within a 0.13dB range. This shows that the fusion architecture is not critical.

Throughout the rest of the paper, we shall use 41 × 41 patches and a search volume of 41 × 41 pixels and 15 frames as the parameters for the patch search. For the fusion network we will use the DnCNN architecture.

Discussion

Surprisingly large patches

An immediate question is why do such big patches achieve such a good performance. On one hand, increasing the patch size has the effect of reducing the variance of the noise in the patch distance, resulting in matches that are less affected by it. This explains the details recovered in the background in Figures 5.4 and 5.5. The large size allow to reliably match the background patches even if the texture seems to be completely obfuscated by noise.

On the other hand, a large patch size becomes a disadvantage for objects with a nontranslational motion. In these cases, finding similar matches in the neighboring frames becomes less likely for larger patches. This can be seen in Figure 5.2. The person in the foreground is rotating as she performs a backflip jump. Patches in the foreground object have similar matches only for the closest neighboring frames (e.g. the face cannot be reconstructed for t ± 4), whereas patches in the background can be matched throughout all the temporal extend of the search region. As a consequence, different pixels will have a different number of relevant non-local features. The fusion network learns during training to be robust to these bad matches. It seems to be able to identify when the patch search fails, relying only on the non-local features that are correlated with the target pixel, and adapting accordingly the amounts of spatial denoising and non-local denoising.

This phenomenon can be observed in Figure 5.6, where we compare the results of the proposed VNLnet against the single frame DnCNN from [ZZC + 17b] and a Non-Local Pixel Mean (NLPM), which simply averages the values of the non-local features given by the patch-matching (please ignore for the moment the column labeled video-DnCNN). On the first two rows, the motion is consistent on the whole crop, and thus the output of NLPM is sharp, indicating good matches. As a result, VNLnet's output shows more details than the single frame DnCNN. However the Non-Local Pixel Mean output is blurry for the person in the third row and the background of the fourth row. For these regions, VNLnet and DnCNN both have similar quality. Meanwhile, for the regions of these two crops with good matches (the background of the third row, and the person of the fourth row), the quality is improved. Using bigger patches will increase the number of patches covering regions of conflicting motion. As a result, we see that the performance gain from 31 × 31 to 41 × 41 is rather small.

A note on motion handling

To evaluate the effectiveness of the non-local features we also train a network with the same architecture, but instead of feeding it with the n non-local features we feed it with the stack of the n neighboring frames. We call this network video-DnCNN. We do this for different values of n, and show the results obtained on Table 5.3. The performance of the video-DnCNN network increases with the number of frames, although less than the proposed VNLnet (see Table 5.1b). In particular, for video-DnCNN the average PSNR on the validation set stagnates at 11 frames. The reason for this is that while the denoising performance increases on sequences with majority of small and smooth motions, it drops significantly when there are many large or irregular motions.

Without the non-local patch-search module, the network has to learn to handle motion implicitly, which makes the task significantly harder. As the number of input frames increases, so does the complexity of the internal motion compensation the network has to learn to denoise accurately. The video-DnCNN network then overfits to the most frequent motion patterns in the training set, and fails when it encounters a different motion. By factoring out the motion, the non-local patch search module removes the need for the network to learn to adapt to various types of motion, enabling a better generalization on various moving scenes.

This can be seen in Figure 5.6, by comparing the results of video-DnCNN and VNLnet, for objects with fast/irregular motion patterns. VNLnet is able to recover much more details, thanks to the patch search.

Figure 5.7 shows the PSNR gain of VNLnet over video-DnCNN for each sequence on the grayscale DAVIS test set. The gain given by the non-local patch-search module is significant, except only for two sequences. These feature fixed cameras and static backgrounds covering most of the frame. The sequences with larger gains have complex motion.

Texture is also an important aspect, as patches with a distinct and highly contrasted texture can be matched more reliably. In Figure 5.8 we plot the per patch PSNR gain of VNLnet against the video-DnCNN method and a single frame DnCNN, in terms of a measure of patch "texturedness". We measure patch texturedness as the sum of the squared gradient magnitudes in a patch:

T (x, t) = h ∇u(x + h, t) 2
(5.4)

where h varies in a patch centered at x, ∇u is spatial gradient of u(•, t) and • is the Euclidean norm. The texturedness is computed on the clean video. We denoised the grayscale DAVIS test set with σ = 20, and for each 41 × 41 patch, we compute its texturedness T (x, t) and the PSNR gain, which we denote by G(x, t). We bin these quantities in a 2D histogram with 400×400 bins which is illustrated by the grayscale image in Figure 5.8 (darker pixels correspond to bins with higher frequency). The figure also shows in red the mean PSNR gain for each level of texturedness, and in blue the same mean, but multiplied by the relative frequency of each texturedness level in the dataset.

The figures confirm that the PSNR gain with respect to both video-DnCNN and single image DnCNN increases with patch texturedness. As expected, the gain is larger with respect to the single frame DnCNN. For patches with very low texturedness the video-DnCNN performs better than the VNLnet. This makes sense, as for those patches the matching will be influenced by the noise. A similar behavior has been observed for patch-based methods in global video denoising [START_REF] Ehret | Global patch search boosts video denoising[END_REF] and external denoising [START_REF] Mosseri | Combining the power of internal and external denoising[END_REF]. Such patches are relatively rare in the test set, which explains the overall positive gains in Figure 5.7.

A related question is whether better results could be obtained replacing the patch search by a frame matching using a standard optical flow. The results of Table 5.1a highlight the importance of very reliable matches, and thus the optical flow would have to be chosen with care. In [START_REF] Tassano | Fastdvdnet: Towards real-time video denoising without explicit motion estimation[END_REF], the authors of DVDnet [START_REF] Tassano | Dvdnet: A fast network for deep video denoising[END_REF] stressed the difficulty of finding a fast and reliable optical flow, and moved away from it for FastDVDnet [START_REF] Tassano | Fastdvdnet: Towards real-time video denoising without explicit motion estimation[END_REF]. In [XCW + 19], the optical flow is computed with a reduced version of SpyNet [START_REF] Ranjan | Optical flow estimation using a spatial pyramid network[END_REF], which is trained together with the rest of the network. In our case, the patch search module is not trainable, and the network is trained to process its output.

Comparison with other methods

In this section, we compare the proposed method VNLnet to VBM3D [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF], VNLB [START_REF] Arias | Towards a bayesian video denoising method[END_REF], and DnCNN [ZZC + 17b] for grayscale videos, and VBM3D [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF], VNLB [START_REF] Arias | Towards a bayesian video denoising method[END_REF], DnCNN [ZZC + 17b], ViDeNN-G [START_REF] Claus | Videnn: Deep blind video denoising[END_REF], DVDnet [START_REF] Tassano | Dvdnet: A fast network for deep video denoising[END_REF], and FastDVDnet [START_REF] Tassano | Fastdvdnet: Towards real-time video denoising without explicit motion estimation[END_REF] for color videos. DnCNN was applied frame-by-frame.

We trained grayscale and color networks for AWGN of standard deviation 10, 20 and 40. To highlight the fact that a CNN method can be easily re-targeted to different noise distributions, we also trained a grayscale network for Gaussian noise correlated by a 3 × 3 box kernel such that the final standard deviation is σ = 20, and 25% uniform Salt and Pepper noise (removed pixels are replaced by random uniform noise).

Table 5.4 shows the denoising results obtained on the two compared datasets. For grayscale videos, we also include results for SPTWO [START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF] and VBM4D [START_REF] Maggioni | Video Denoising Using Separable 4D Nonlocal Spatiotemporal Transforms[END_REF], computed in [START_REF] Arias | A comparison of patchbased models in video denoising[END_REF] for the DERF dataset. Figures 5.9 and 5.10 show results for the most relevant RGB methods. VNLB (Video Non-Local Bayes) outperformed on average all other methods on the DERF dataset. Meanwhile on the DAVIS dataset, our method performed the best both in grayscale and color, for all the three tested noise levels. VNLB ranked second, except in color for high noise levels, where it was surpassed by DVDnet and FastDVDnet.

A comparison of the results for grayscale and color in Table 5.4 reveals that CNN-based methods exploit better the correlations between color channels: while for grayscale, VBM3D significantly outperforms DnCNN in PSNR on the DAVIS dataset, the reverse occurs for color. Moreover, VNLnet performed proportionally better in color than in grayscale. This should not come as a surprise, since the way in which VBM3D and VNLB treat color is rather heuristic: an orthogonal color transform is applied to the video which is supposed to decorrelate color information. Then the processing of each color channel of a group of patches is done independently.

In order to better compare qualitative aspects of the results we show some details in Figures 5.9 and 5.10. For some scenes, VNLnet recovers significantly more details in the background, as shown in Figure 5.9. In general, we observe that VNLnet, and the other video CNN methods (ViDeNN-G, DVDnet, and FastDVDnet) have better background reconstruction than VNLB. This can be seen in Figure 5.9 and Figure 5.10. Some details however are better recovered by VNLB and VNLnet. For example in Figure 5.10 both methods recover the red lights in the top left corner of the image in the first column, while for the three other methods the lights do not appear. In the second column, VNLB does not reconstruct the trees as well as the CNN-based methods, but manages to recover the color of the clothes of the person in the bottom left. VNLnet not only recovers better the tree structure, but also recovers the clothes correctly. None of the methods restore satisfyingly the grass texture in the third column of Figure 5.10 for the tested noise level. This highlights that there is room for improvement. All five methods achieve reasonable temporal consistency, which is an important quality requirement for video denoising.

One of the benefits of CNNs over traditional model-based approaches is that they can be easily targeted to handle other noise distributions by simply re-training them. We demonstrate this by reporting the results of our method on salt-and-pepper noise and correlated noise in Table 5.4. We include the result of VNLB as a reference.

GRAYSCALE

In summary, the proposed approach for video denoising obtains state-of-the-art results on both test sets. In particular, it outperforms previous CNN approaches. In light of this, we can conclude that effectively exploiting a large number of surrounding frames is key. Indeed, the proposed VNLnet uses 15 frames, in comparison to the 3 frames used by ViDeNN-G and 5 frames of DVDnet and FastDVDnet. Most of these methods avoid relying on an explicit optical flow computation, which can be unreliable given that the input frames are noisy. FastDVDnet and ViDeNN-G do so by performing an early fusion of triplets of consecutive frames without alignment. The non-local features computed via patch correspondences result in an effective way to present the information of large frame neighborhoods to a network that merges them. Training the fusion network is then straightforward.

Running times and number of parameters

In Tables 5.5 and 5.6, we compare the running time in grayscale and color of several methods when denoising a video frame. As before, we consider DnCNN as the fusion network. In Table 5.5, the compared methods are run on a single CPU core, while in Table 5.6, a system with an Intel Xeon W-2145 and a NVIDIA TITAN V is used. For both tables, the loading and writing time of the videos were subtracted. Since we do not have a CPU implementation of the patch search layer, we cannot measure a CPU time for VNLnet. On the GPU, for grayscale videos of 960 × 540 the non-local search takes 822ms, where as the DnCNN fusion network takes 95ms. Extrapolating this, we could expect a CPU time 8 to 9 times slower than DnCNN.

Most of the running time of our method is spent in the patch search. Our GPU implementation of patch search is similar to the convolution-based patch search described in [START_REF] Davy | GPU acceleration of NL-means, BM3D and VBM3D[END_REF]. With the default parameters, the non-local search is costly because matches are searched in 15 frames for patches centered at every pixel of our image. These parameters can be modified, trading off speed by denoising quality. In the Tables 5.7 we show how the time spent in the patch search as a function of the patch size, spatial search region size and number of frames. These behave as expected: the time grows linearly with the number of frames, and quadratically with respect to the patch and search sizes. For example, with a search region of 21 × 21, the per frame patch search time is reduced by more than three times, while the drop in PSNR is only 0.16dB (see Table 5 The table also shows the patch search time normalized by the number of frames that need to be aligned in the search region. The implementation could be further accelerated by reducing the size of the 3D window using tricks explored in other papers. VBM3D for example centers the search on each frame on small windows around the best matches found in the previous frame. A related acceleration is to use a search strategy based on PatchMatch [START_REF] Barnes | Patchmatch: A randomized correspondence algorithm for structural image editing[END_REF].

One of the benefits of the proposed approach is that the fusion network can be simple, resulting in an easier training and a smaller memory footprint of the network weights. The three tested architectures were designed for single image denoising. This reduces the number of trainable parameters, in comparison with other video processing networks. The DnCNN and EDSR networks have both around 0.56M parameters, while the U-Net has 1M. For RGB videos, our DnCNN with 25 layers and 96 channels has 1.9M parameters, roughly the same as, FastDVDnet [START_REF] Tassano | Fastdvdnet: Towards real-time video denoising without explicit motion estimation[END_REF], which requires 2M parameters as it uses two U-Nets.

Other works such as [XCW + 19, TDV19a] use an optical flow sub-network for aligning the frames. This has the advantage that the alignment can be trained together with the fusion network [XCW + 19]. However, the smallest current optical flow networks have between 1M and 5M parameters [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF], and take around 50ms to 100ms (depending on the desired quality) to estimate the optical flow between a pair of frames.

Conclusions

We described an effective way of incorporating temporal non-local information into a CNN for video denoising. The proposed method computes for each image patch the n most similar neighbors on a spatio-temporal window and gathers their central pixels to form a non-local feature vector which is fed to a CNN. Our method yields a significant gain compared to other CNN approaches. It has similar performance to the best non-CNN method evaluated, VNLB, outperforming it on the largest of our test datasets. In addition, we noted that CNN approaches tend to better reconstruct backgrounds than VNLB, which are perceptually relevant areas. To prevent dataset bias we also proposed a public training set comprising 17k videos from 64 different categories and a simulation strategy that emulates different levels of sharpness.

We have seen the importance of reliable matches: On the validation set, the best performing method used patches of size 41 × 41 for the patch search. We have also noticed that on regions with non-reliable matches (complex motion), the network seems to revert to a result similar to single image denoising. Thus we believe future works should focus on improving this area. Table 5.7: Time spent in the patch search, for different values of the parameters. We report both the average total time spent in the search (considering all frames in the search region) and the time normalized by the depth of the search region (minus the central frame for which there is no search).

6 Model-blind video denoising via frame-to-frame training

In the previous chapters, the noise model is always considered fixed and known. However, modeling the processing chain that has produced a video is a difficult reverse engineering task, even when information on the camera is available. This makes model based video processing still a more complex task. In this chapter we propose a fully blind video denoising method, with two versions batch and online. This is achieved by fine-tuning a pretrained AWGN image denoising network to the video with a novel frame-to-frame training strategy. Our denoiser can be used without knowledge of the origin of the video and the post-processing steps applied from the camera sensor. The on-line process only requires a couple of frames before achieving visually pleasing results for a wide range of perturbations. It nonetheless reaches state-of-the-art performance for standard Gaussian noise, and can be used offline with the batch fine-tuning for even better performance. This work has been published in [EDM + 19].

Introduction

As As we have seen in the previous chapters, the most widely adopted assumption in the literature is that noise is additive white Gaussian (AWGN). This is justified by the fact that the noise generated by the photon count process at the imaging sensor can be modeled as Poisson noise, which in turn can be approximated by AWGN after a variance stabilizing transform (VST) [START_REF] Francis | The transformation of poisson, binomial and negativebinomial data[END_REF][START_REF] Makitalo | Optimal inversion of the anscombe transformation in low-count poisson image denoising[END_REF][START_REF] Makitalo | A closed-form approximation of the exact unbiased inverse of the anscombe variance-stabilizing transformation[END_REF]. However, in many practical applications the data available is not the raw data straight from the sensor. The camera output is the result of a processing pipeline, which can include quantization, demosaicking, gamma correction, compression, etc. The noise at the end of the pipeline is spatially correlated and signal dependent, and it is difficult to model. Furthermore the details of the processes undergone by an image or video are usually unknown. To make things even more difficult, a large amount of images and videos are generated by mobile phone applications which apply their own processing of the data (for example compression, filters, or effects selected by the user). The specifics of this processing are unknown, and might change with different releases.

The literature addressing the case of an unknown noise model is much more limited. The works [LCM15, GPMA18] address denoising noisy compressed images. RF3D [START_REF] Maggioni | Joint removal of random and fixed-pattern noise through spatiotemporal video filtering[END_REF] handles correlated noise in infrared videos. Data-driven approaches provide an interesting alternative when modeling is not challenging. CNNs have been applied successfully to denoise images with non-Gaussian noise [ZZC + 17a, CCXK18, GYZ + 18]. In applications in which the noise type is unknown, one could use model-blind networks such as DnCNN-3 [ZZC + 17a] trained to denoise several types of noise, or the blind denoiser of [GYZ + 18]. These however have two important limitations. First, the performance of such model-blind denoising networks very often drops with respect to model-specific networks [ZZC + 17a]. Second, training the network requires a dataset of images corrupted with each type of noise that we wish to remove (or the ability to generate it synthetically [GYZ + 18]). However, generating ground truth data for real photographs is not straightforward [START_REF] Plotz | Benchmarking denoising algorithms with real photographs[END_REF][START_REF] Chen | Learning to see in the dark[END_REF]. Furthermore, in many occasions we do not have access to the camera, and a single image or a video is all that we have.

In this chapter we show that, for certain kinds of noise, in the context of video denoising one video is enough: a network can be trained from a single noisy video by considering the video itself as a dataset. Our approach is inspired by two works: the one-shot object segmentation method [CMPT + 17] and the noise-to-noise training proposed in the context of denoising by [LMH + 18]. + 17] suggested a oneshot framework for object segmentation in video, where an object is manually segmented on the first frame and the objective is to segment it in the rest of the frames. Their main contribution is the use of a pre-trained classification network, which is fine-tuned to a manual segmentation of the first frame. This fine-tuned network is then able to segment the object in the rest of the frames. This generalizes the one-shot principle from classification to other types of problems. Borrowing the concept from [CMPT + 17], our work can be interpreted as a one-shot blind video denoising method: a network can denoise an unseen noise type by fine-tuning it to a single video. In our case, however, we do not require "labels" (i.e. the ground truth images without noise). Instead, we benefit from the noise-to-noise training proposed by [LMH + 18]: a denoising network can be trained by penalizing the loss between the predicted output given a noisy and a second noisy version of the same image, with an independent realization of the noise. We benefit from the temporal redundancy of videos and use the noise-to-noise training between adjacent frames to fine-tune a pre-trained denoising network. That is, the network is trained by minimizing the error between the predicted frame and the past (or future) frame. The noise used to pre-train the network can be very different from the type of noise in the video.

The aim of one-shot learning is to train a classifier network to classify a new class with only a very limited amount of labeled examples. Recently Caelles et al. [CMPT

We present the different tools, namely one of the state-of-the-art denoising network DnCNN [ZZC + 17a] and a training principle for denoising called noise2noise [LMH + 18], necessary to derive our refined model in Section 6.2. We present our truly blind denoising principle in Section 6.3. We compare the quality of our blind denoiser to the state of the art in Section 6.4. Finally we conclude and open new perspectives for this type of denoising in Section 6.5.

Preliminaries

The proposed model-blind denoiser builds upon DnCNN and the noise-to-noise training. In this section we provide a brief review of these works, plus some other related work.

DnCNN

DnCNN [ZZC + 17a] was the first neural network to report a significant improvement over patchbased methods such as BM3D [START_REF] Dabov | Image denoising with block-matching and 3D filtering[END_REF] and WNNM [START_REF] Gu | Weighted nuclear norm minimization with application to image denoising[END_REF]. It has a simple architecture consisting of 17 convolutional layers. The first layer consists of 64 3×3 followed by ReLU activations and outputs 64 feature maps. The next 15 layers also compute 64 3 × 3 convolutions, followed by batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] and ReLU. The output layer is simply a 3 × 3 convolutional layer.

To improve training, in addition to the batch normalization layers, DnCNN uses residual learning, which means that network is trained to predict the noise in the input image instead of the clean image. The intuition behind this is that if the mapping from the noisy input f to the clean target u is close to the identity function, then it is easier for the network to learn the residual mapping, f → f -u.

DnCNN provides state-of-the-art image denoising for Gaussian noise with a rather simple architecture. For this reason we will use it for all our experiments.

Noise-to-noise training

The usual approach for training a neural network for denoising (or other image restoration problems) is to synthesize a degraded image f i from a clean one u i according to a noise model. Training is then achieved by minimizing the empirical risk which penalizes the loss between the network prediction F θ (f i ) and the clean target u i . This method cannot be applied for many practical cases where the noise model is not known. In these settings, noise cannot be synthetically added to a clean image. One can generate noisy data by acquiring it (for example by taking pictures with a camera), but the corresponding clean targets are unknown, or are hard to acquire [START_REF] Chen | Image blind denoising with generative adversarial network based noise modeling[END_REF][START_REF] Plotz | Benchmarking denoising algorithms with real photographs[END_REF].

Lehtinen et al. [LMH + 18] recently pointed out that for certain types of noise it is possible to train a denoising network from pairs of noisy images (f i , g i ) corresponding to the same clean underlying data and independent noise realizations, thus eliminating the need for clean data. This allows learning networks for noise that cannot be easily modeled (an appropriate choice of the loss is still necessary though so that the network converges to a good denoising).

Assume that the pairs (f, u) are distributed according to p(f, u) = p(u|f )p(f ). For a dataset of infinite size, the empirical risk of an estimator F converges to the Bayesian risk, i.e. the expected loss (law of large numbers): R(F) = E f,u { (F(u), f )}. The optimal estimator F * depends on the choice of the loss. From Bayesian estimation theory [START_REF] Kay | Fundamentals of statistical processing, volume i: Estimation theory: Estimation theory[END_REF] we know that:1

= L 2 ⇒ F * (f ) = E{u|f } (6.1) = L 1 ⇒ F * (f ) = median{u|f } (6.2) = L 0 ⇒ F * (f ) ≈ mode{u|f } (6.3)
Here E{u|f } denotes by the expectation of the posterior distribution p(u|f ) given the noisy observation f . During training, the network learns to approximate the mapping f → F * (f ).

The key observation leading to noise-to-noise training is that the same optimal estimators apply when the loss is computed between F(f ) and g, a second noisy version of u. In this case we obtain the respectively the mean, median or mode of the posterior p(g|f ). Then, for example if the noise is such that E{g|f } = E{u|f }, then the network can be trained by minimizing the MSE loss between F (f ) and a second noisy observation g. If the median (resp. the mode) is preserved by the noise, then the L 1 loss (resp. the L 0 ) loss can be used. The network F is then trained on pairs of realizations of the same image (x 1 i , x 2 i ), using the self-supervised loss

L = (x 1 i ,x 2 i ) f (x 1 i ) -x 2 i p .

Model-blind video denoising

In this section we show how one can use a pre-trained denoising network learned for an arbitrary noise and fine-tune it to other target noise types using a single video sequence, attaining the same performance as a network trained specifically for the target noise for classic noise. This fine tuning can be done with batches (using the whole video as a dataset) or online, i.e. frame-byframe, depending on the application and the computational resources at hand. As we will show in Section 6.4, starting from a pre-trained network is key for the success of the proposed training, as we do not have a large dataset available as in [LMH + 18], but only a single video sequence. The use of a pre-trained network is in part motivated by works on transfer learning such as Zamir et al. [ZSS + 18]. Denoising different noise models are related tasks. Our intuition is that a part of the network focuses on the noise type while the rest encodes features of natural images.

Our approach is inspired by the one-shot video object segmentation approach of [CMPT + 17], where a classification network is fine-tuned using the manually segmented first frame, and then applied to the other frames. As opposed to the segmentation problem, we do not assume that we have a ground truth (clean frames). Instead, we adapt the noise-to-noise training to a single video.

We need pairs of independent noisy observations of the same underlying clean image. For that we take advantage of the temporal redundancy in videos: we consider consecutive frames as observations of the same underlying clean signal transformed by the motion in the scene. To account for the motion we need to estimate it and warp one frame to the other. We estimate the motion using an optical flow. We use the TV-L1 optical flow [START_REF] Zach | A duality based approach for realtime tv-l 1 optical flow[END_REF] with an implementation available in [START_REF] Sánchez Pérez | TV-L1 Optical Flow Estimation[END_REF]. This method is reasonably fast and is quite robust to noise when the flow is computed at a coarser scale.

Let us denote by o t the optical flow from frame f t to frame f t-1 . The warped f t-1 is then

f w t-1 (x) = f t-1 (x + o t (x)
) (we use bilinear interpolation). Similarly, we define the warped clean frame u w t-1 . We assume (i) that the warped clean frame u w t-1 matches u t , i.e. u t (x) ≈ u w t-1 (x), and

(ii) that the noise of consecutive frames is independent.

Occluded pixels in the backward flow from t to t -1 do not have a correspondence in frame t -1. Nevertheless, the optical flow assigns them a value. We use a simple occlusion detector to eliminate these false correspondences from our loss. A simple way to detect occlusions is to determine regions where the divergence of the optical flow is larger than a threshold τ . This idea and how τ is computed is presented in [START_REF] Buades | Patch-based video denoising with optical flow estimation[END_REF]. We therefore define a binary occlusion mask as

κ t (x) = 0 if |div o t (x)| > τ 1 if |div o t (x)| ≤ τ. (6.4)
Pixels with an optical flow that points out of the image domain are considered occluded. In practice, we compute a more conservative occlusion mask by dilating the result of Eq. (6.4).

We then compute the loss masking out occluded pixels. For example, for the L 1 loss we have:

1 (f, g, κ) = x κ(x) |f (x) -g(x)| . (6.5)
Similarly one can define masked versions of other losses. In the noise-to-noise setting, the choice of the loss depends on the properties of the noise [LMH + 18]. The noise types that can be handled by each loss in noise-to-noise have a precise characterization (the mean/median/mode of the noisy posterior p(g|f ) have to be equal to those of the clean posterior p(u|f )). Verifying this requires some knowledge about noise distribution. In the absence of such knowledge, since the method is reasonably fast, an alternative would be to test different losses and see which one gives the best result.

In principle our method is able to deal with the same noise types as noise-to-noise. In practice we have some limitation imposed by the registration as it degrades for severe noise. For this reason we do not show examples with non-median preserving noise requiring the L 0 loss. For all our experiments we use the masked L 1 loss since it has better training properties than the L 2 [START_REF] Zhao | Loss Functions for Image Restoration With Neural Networks[END_REF]. Most relevant noise types often encountered in practice (Poisson, jpeg-compressed, low-freq. noise) can be handled by the L 1 loss and the registration.

We now have pairs of images (f t , f w t 1 ) and the corresponding occlusion masks κ t and we apply the noise-to-noise principle to fine-tune the network on this dataset. In order to increase the number of training samples the symmetric warping can also be done, i.e. warping f t+1 to f t using the forward optical flow from f t to f t+1 . This allows to double the amount of data used for the fine-tuning. We consider two settings: batch and online training.

Batch fine-tuning. We denote the network as a parametrized function F θ , where θ is the parameter vector. In the batch setting we fine-tune the network parameters θ by doing a fixed number N of steps of the minimization of the masked loss over all frames in the video:

θ ft = N,θ 0 arg min θ T t=1 1 (F θ (f t ), f w t-1 , κ t ) (6.6)
where by N,θ 0 arg min θ E(θ) we denote an operator which does N optimization steps of function E starting from θ 0 and following a given optimization algorithm (for instance gradient descent, Adam [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], etc.). The initial condition for the optimization is the parameter vector of the pre-trained network. The fine-tuned network is then applied to the rest of the video.

On-line fine-tuning In the on-line setting we train the network in a frame-by-frame fashion.

As a consequence we denoise each frame with a different parameter vector θ ft t . At frame t we compute θ ft t by doing N optimization steps corresponding to the minimization of the loss between frames t and t -1:

θ ft t = N,θ ft t-1 arg min θ 1 (F θ (f t ), f w t-1 , κ t ). (6.7)
The initial condition for this iteration is given by the fine-tuned parameter vector at the previous frame θ ft t . The first frame is denoised using the pre-trained network. The fine-tuning starts for the second frame. A reasonable concern is that the network overfits the given realization of the noise and the frame at each step. This is indeed the case if we use a large number of optimization iterations N at a single frame. A similar behavior is reported in [START_REF] Ulyanov | Deep image prior[END_REF], which trains a network to minimize the loss on a single data point. We prevent this from happening by using a small number of iterations (e.g. N = 20). We have observed that the parameters fine-tuned at t can be applied to denoise any other frame without any significant drop in performance.

The on-line fine-tuning addresses the problem of lifelong learning [ZSS + 18] by continuously adapting the network to changes in the distribution of noise and signal. This is particularly useful when the statistics of the noise depend on time-varying parameters (such as imaging sensors affected by temperature).

Experiments

In this section we demonstrate the flexibility of the proposed fine-tuning blind denoising approach with several experimental results. For all these experiments the starting point for the fine-tuning process is a DnCNN network trained for an additive white Gaussian noise of standard deviation σ = 25. In all cases we use the same hyper-parameters for the fine tuning: a learning rate of 5.10 -5 and N = 20 iterations of the Adam optimizer. For the batch case we use the entire video. The videos used in this section come from Derf's database2 . They've been converted to grayscale by averaging the three color channels and downscaled by a factor two in each direction to ensure that they contain little to no noise. The code and data to reproduce the results presented in this section are available on https://github.com/tehret/blind-denoising.

To the best of our knowledge there is not any other blind video denoising method in the literature. We will compare with state-of-the-art methods on different types of noise. Most methods have been crafted (or trained) for a specific noise model and often a specific noise level. We will also compare with the only available blind denoising method we found, an image denoising method proposed by Lebrun et al. [START_REF] Lebrun | The noise clinic: a blind image denoising algorithm[END_REF] which assumes a Gaussian noise model with variance depending on the intensity and the local frequency of the image. This model was proposed for denoising of compressed noisy images. We cannot compare with some more recent blind denoising methods, such as [START_REF] Chen | Image blind denoising with generative adversarial network based noise modeling[END_REF], because there is no code available. We will also compare with DnCNN [ZZC + 17a] and VBM3D [START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF]. VBM3D is a video denoising algorithm. All the other methods are image denoising applied frame-by-frame (perspectives for videos are mentioned in Section 6.5).

The goal of the first experiment is to compare against reference networks trained for these noises the regular way. The per-frame PSNRs are presented in Figure 6.2. We applied the proposed learning process to a sequence contaminated with AWGN with standard deviation σ = 25, which is precisely the type of noise the network was trained on and verified that it does not deteriorate the pre-training. The batch fine-tuning performs on par with the pre-trained network. The PSNR of the on-line process has a higher variance, with some significant drops for some frames. For σ = 50, we can see that both fine-tuned networks perform better than the pre-trained network for σ = 25. In fact their performance is as good as the DnCNN network trained specifically for σ = 50 (actually the batch trained performs even slightly better than the reference network). Our process also outperforms the "noise clinic" of [START_REF] Lebrun | The noise clinic: a blind image denoising algorithm[END_REF].

We have also tested the proposed fine-tuning on four other types of noise: multiplicative Gaussian, correlated, salt and pepper and compressed Gaussian. We present the corresponding per-frame PSNRs in Figure 6.3. The multiplicative Gaussian noise is given by

f t (x) = u t (x) + r t (x)u t (x), (6.8)
where r t (x) is white Gaussian noise with standard deviation of 75/255 (the images are within the range [0,1]). The resulting variance σ 2 t (x) depends on the pixel intensity u t (x). The correlated noise is obtained by convolving AWGN with a disk kernel. The resulting standard deviation is σ = 25. The salt and pepper uniform noise is like the one used [LMH + 18], obtained by replacing with probability 0.25 the value of a pixel with a value sampled uniformly in [0, 1]. Finally, the compressed Gaussian noise, results from compressing an image corrupted by an AWGN of σ = 25 with JPEG. The last one is particularly interesting because it is a realistic use case for which the noise model is quite hard to estimate [START_REF] Gonzalez | Joint denoising and decompression using cnn regularization[END_REF]. While in this case the noise can be generated synthetically for training a network over a dataset, this is not possible with other compression tools (for example for proprietary technologies). We can see the effectiveness of the fine-tuning for all examples. The batch training is more stable (smaller variance) and gives slightly better results, although the difference is small.

A visual comparison with other methods is shown in Figure 6.4 for JPEG compressed noise and in Figure 6.5 for AWGN with σ = 50. Visual examples on real data are presented in the supplementary material. The result of the fine-tuned network has no visible artifacts and is visually pleasing even though the network has never seen this type of noise before the fine-tuning. A limitation of the method is the oversmoothing of texture. Indeed DnCNN has a tendency of oversmoothing textures. Using a network designed for video denoising should help recover more texture and improve temporal consistency [DEM + 19]. Another cause is the optical flow. Since it is computed on downscaled noisy frames it is imprecise around edges. This leads to false correspondences between frames and introduces some blur. Improved registration should lead to sharper results.

In Tables 6.1 and 6.2 we show the PSNR of the results obtained on 4 sequences for AWGN of σ = 50 and JPEG compressed AWGN of σ = 25 and compression factor 10. For the case of AWGN the fine-tuned networks attain the performance of the DnCNN trained for that specific noise. For JPEG compressed Gaussian noise, the batch fine-tuned network is on average 0.3dB above the pre-trained network. require more frames for convergence. In turn the result has smaller variance. A similar analysis can be done for the learning rate. We also show the importance of using a pre-trained network compared to a random initialization. There is a 2dB gap in favor of the pre-trained network.

The other important parameter is the number of frames used for fine-tuning. The fine-tuning is stopped at a frame t 0 and θ ft t 0 is used to process the remaining frame. We can see that the more frames used for the fine-tuning the better the performance.

Finally Figure 6.7 shows examples of lifelong learning. The first example shows a slowly evolving noise (starting with a Gaussian noise with standard deviation σ = 25 that linearly increases up to σ = 50). The fine-tuned network performs better than the two reference networks for respectively σ = 25 and σ = 50. The second example shows a sudden change (starting with a Gaussian noise with standard deviation σ = 50 and changes to Salt and pepper noise at frame 200). In that case the fine-tuned network adapts quickly to the new noise model.

The running time depends on the network. We used DnCNN but other networks can be used instead and trained with the proposed method. Each fine-tuning iteration runs a back-propagation step which takes 0.33s on a NVIDIA Titan XP for DnCNN for a 960 × 540 frame. Fifty frames with 20 iterations per frame take 5 mins. For comparison, training DnCNN from scratch over a dataset requires around 6h (and a dataset). By using a lighter network and reducing the perframe-iterations it might be possible to achieve real time frame rates. Moreover, the fine-tuning can be done on a fixed number of frames at the beginning or run in the background each number of frames for cases when the computational efficiency is important.

A challenging test case of our blind denoising is old digitalized films. The difficulty with this data is that the film quality degraded gradually with time and can be physically damaged during its manipulation and reproduction, creating several type of artifacts. The two examples shown in Figures 6.8, 6.9, and 6.10 are examples from footage of World War I3 . In addition to the noise and the damaged parts of the film, there's also a strong compression that has been applied after digitalization. All of this makes modelling the noise very difficult. Yet, the proposed frame-to- frame fine-tuning strategy is still able to learn to denoise these sequences. The blind denoiser is able to remove most of what can be consider as noise while retaining most details. In Figure 6.9 we show a comparison with the pre-trained network (starting point of the fine-tuning) and the result of VBM3D. VBM3D receives as input the noise level σ. We tested several noise levels and chose then one the seemed best. As one can see in Figure 6.9 the blind denoiser keeps more details in the fields, the building or the airplane than the pre-trained network and VBM3D.

Our last experiment is with a video shot with a Samsung Galaxy S7. The video is shot in a low light, and processed by the camera pipeline. This means that it has been demosaicked, denoised (by a fast method running directly on the phone), among other quality enhancement algorithms, and finally compressed. The remaining noise is therefore completely distorted, being colored and non-stationary. We used a pretrained network which is a color DnCNN trained for Gaussian noise with standard deviation σ = 25. We use these hyper-parameters for the fine tuning: a learning rate of 1.10 -4 and N = 10 iterations of the Adam optimizer. Figure 6.11 presents a crop of the video. Here again, blind denoising largely removes the artifacts left by the phone's pipeline and therefore improves the overall visual quality of the video.

Discussion and perspectives

Denoising methods based on deep learning often require large datasets to achieve state-of-theart performance. Lehtinen et al. [LMH + 18] pointed out that in many cases the clean ground truth images are not necessary, thus simplifying the acquisition of the training datasets. With the framework presented in this chapter we take a step further and show that a single video is often enough, removing the need for a dataset of images. By applying a simple frame-to-frame training on a generic pre-trained network (for example a DnCNN network trained for additive Gaussian noise with fixed standard deviation), we successfully denoise a wide range of different noise models even though the network has never seen the video nor the noise model before its fine-tuning. This opens the possibility to easily process data from any unknown origin.

Future works will focus on improving the denoising quality and computation speed. First, given that the application is video denoising, it is expected that better results will be achieved by a video denoising network (the DnCNN network processes each frame independent of the others). Using the temporal information could improve the denoising quality, just like video denoising methods improve over frame-by-frame image denoising methods, but also might stabilize the variance of the result for the on-line fine-tuning. Recent works have also shown the possibility of real time fine-tuning [TRJ + 19, TTP + 19] paving the way to improve computation time.

Demosaicking with deep learning

Most cameras capture the information of only one color for a given pixel. This results in a mosaicked image that must be interpolated to get three colors at each pixel. The step going from a mosaicked image to a regular RGB image is called demosaicking. This chapter studies two recent demosaicking methods based on convolutional neural networks that achieve artifact-free state-of-the-art results: Deep joint demosaicking and denoising by Gharbi et al.and Color image demosaicking via deep residual learning by Tan et al.. We show that these methods beat by almost two decibels the best human-crafted methods, while being faster by one order of magnitude. The review presented in this chapter motivates the need for the self-supervised learning presented in the next chapter.

Introduction

Most cameras capture only one color per pixel, this color being determined by a color filter array (CFA) located on top of the sensor. The most commonly used CFA is the so-called Bayer pattern, consisting of a regular subsampling of each color channel. This means that each pixel of the resulting raw image contains one third of the necessary information, and that the color channels are sampled on different grids. The problem of interpolating the missing color channels at each pixel, by using the information available in a neighborhood, is called demosaicking. It is a challenging ill-posed inverse problem.

The simplest demosaicking method is the independent bilinear interpolation of each channel. Yet recent methods are far more sophisticated. Getreuer [START_REF] Getreuer | Image Demosaicking with Contour Stencils[END_REF] solves the demosaicking by using contours of objects as guide. Mairal et al. [MBP + 09] suggested combining group sparsity and dictionary learning to achieve optimal performance. In a series of paper, Kiku et al. [START_REF] Kiku | Residual interpolation for color image demosaicking[END_REF], [START_REF] Kiku | Minimized-laplacian residual interpolation for color image demosaicking[END_REF] and [START_REF] Monno | Adaptive residual interpolation for color image demosaicking[END_REF] proposed increasingly complex interpolation methods where they use the interpolated Green channel as a guide for the Red and Blue channels. These last four methods may be considered among the best ones designed by humans. A recently emerged trend consists in using convolutional neural networks for image processing and computer vision applications. See for example the early attempt in [START_REF] Wang | A fast c++ implementation of neural network backpropagation training algorithm: Application to bayesian optimal image demosaicking, image processing on line[END_REF]. These methods have by now reached state-of-the-art results.

The rest of the chapter is organized as follows: In Section 7.2 we present the method "Deep joint demosaicking and denoising" by Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF], in Section 7.3 we present the method "Color image demosaicking via deep residual learning" by Tan et al. [START_REF] Tan | Color image demosaicking via deep residual learning[END_REF]. A quantitative and qualitative comparison with previous state-of-the-arts methods is made in Section 7.4.

Deep joint demosaicking and denoising by Gharbi et al. [GCPD16]

Architecture

The network starts by downsampling the CFA image into a four-channel image that then goes through a serie of 14 Conv+bias layers with 64 features and 3 × 3 convolutions. A 15th layer of Conv+BN+ReLu produces 12 features with 3 × 3 convolutions. It is followed by an upsampling layer producing an RGB image of twice the width and twice the height. The CFA input is split into the respective channels and concatenated to this intermediate RGB image. This layer acts as a residual layer. After this concatenation, a Conv+bias layer is added before the final layer producing the RGB output. The architecture is depicted in Figure 7.1. The architecture is slightly modified when working with noisy or Fujifilm X-Trans data. For X-Trans data, the image is not downsampled. Indeed Gharbi et al.argue that since the X-Trans pattern is 6 × 6 (as illustrated in Figure 7.2), the downsampling would be too aggressive. The layer acting as a residual layer is also removed. For noisy data, the noise level (noise standard deviation) σ is concatenated into the downsampling layer (using a channel with values all equal to σ). The rest of the network stays the same.

Training

Loss A classic L 2 loss is used for training in all three cases (Bayer demosaicking, X-Trans demosaicking and joint demosaicking and denoising). All noise levels are trained at the same time for joint demosaicking and denoising.

Training dataset A first pre-training was done using natural images. In particular 1.3 millions images from ImageNet [DDS + 09] and 1 million images from MirFlickr [START_REF] Mark | The mir flickr retrieval evaluation[END_REF] were chosen for this pre-training. The images chosen had a minimum size of 16M pixels and then were downsampled by a factor of 4. Data augmentation was performed by flipping, rotating and applying a 1 pixel-shift. CFA images were obtained by masking.

Gharbi et al. argue that classic metrics such as L 2 and PSNR are not much impacted by demosaicking artifacts. Moreover, since difficult patches are rare, this means that it would be hard to train a network for demosaicking without artifacts unless the training set be biased towards the creation of such artifacts. For this reason, they created two datasets (one for each type of artifact, namely luminance and moiré) containing only difficult patches used to fine-tune the network. Using the pre-trained network, luminance artifacts are found using the HDR-VDP2 metric [START_REF] Mantiuk | Hdrvdp-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions[END_REF]. Moiré artifacts are found by looking at frequencies that have a much larger energy in Fourier for the demosaicked patch. Figure 7.3 shows examples of patches in these datasets.

Weights initialization He et al.suggested in [HZRS15] a way to generalize the "Xavier" initialization initially proposed by Glorot and Bengio [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF] to take into account ReLUs non-linearity. This allows for a good convergence even for very deep models. He et al.derive conditions on the weights of the kernels at each layer to avoid the problems of vanishing or exploding gradients and vanishing or exploding signal. It leads to initializing all kernels using a zero-mean Gaussian distribution with variance 2 n where n is the size of the kernel.

Training parameters While most parameters were initialized using the method presented in the previous paragraph, biases were initialized to 0. The patch size for the training was set to 128 × 128 and the batch size to 64. The training was done using the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF]. The learning rate was set to 1.10 -4 and an L 2 weight decay to 1.10 -8 . The other parameters for the Adam optimizer were left to their default value. On top of the weight decay, there was a decrease of the learning rate of a factor 10 when the validation error stagnated for ten epochs.

The model took about two to three weeks to train on a NVIDIA Titan X1 .

Color image demosaicking via deep residual learning by Tan et al. [TZZZ17]

Architecture

The network starts with a bilinear interpolation of the CFA image into a first-estimate RGB image that then goes through a series of 6 convolutional layers (the three middle layers also have batchnormalization (BN) [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] and a ReLU) with 64 features and 3 × 3 convolutions (convolutions are padded to ensure that the output has the same size as the input). The bilinearly interpolated image is added to the result of these convolutional layers producing a first estimate RGB image. It then goes through a second series of 5 convolutions (the three middle layers also have BN+ReLU) before adding the first estimate to produce the final result. The architecture is depicted in Figure 7.4.

Training

Loss The network can be decomposed into two steps: The first part (corresponding to the first residual section) estimates the green channel and the second estimates the other two channels (using the estimated green channel as a guide). For this reason the loss is split into two terms: the first term focuses only on the green channel and the second one takes into account the final result. For a training pair (x, y), where x is the CFA image and y the corresponding RGB image, and θ = (θ 1 , θ 2 ) the parameters of the network (respectively the first and second part of the network for θ 1 and θ 2 , the loss is defined as

L(x, y, θ = (θ 1 , θ 2 )) = y G -N θ 1 (x) G 2 2 + N θ (x) -y 2 2 (7.1)
where the G sub-index corresponds to extracting the green channel and N θ 1 is the partial network up to the first residual section.

Training dataset The dataset used for training is the Waterloo dataset presented in [MDW + 17].

A sample of images is shown in Figure 7.5. It is made of 4744 images, out of which 100 are used for validation. Data augmentation was performed by flipping and rotating the images. CFA images were generated by appropriate masking. Finally, out of all training images, 3000 batches of 128 patches were extracted.

Training parameters The network was initialized using the same methods than [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] presented in Section 7.2.2. The patch size for the training was set to 50 × 50 and the batch size to 128. The training was done using the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with default parameters except for the learning rate. The learning rate was set to 2.10 -4 for the beginning of the training and 1.10

Quantitative and qualitative comparison

In this section we compare both quantitatively and qualitatively some of the most efficient demosaicking methods. We used the state-of-the-art methods LSSC [MBP + 09], RI [START_REF] Kiku | Residual interpolation for color image demosaicking[END_REF], ARI [START_REF] Monno | Adaptive residual interpolation for color image demosaicking[END_REF], MLRI [START_REF] Kiku | Minimized-laplacian residual interpolation for color image demosaicking[END_REF] and Getreuer [START_REF] Getreuer | Image Demosaicking with Contour Stencils[END_REF] that are compared in most recent papers and retain a low rank in most. Actually, ARI and LSSC are generally considered the best handcrafted methods, but they are also very complex. . We also added the average PSNR on a dataset made of the images from all three datasets. In order to avoid boundary effects, the images were first padded (with a padding of 48 pixels) using symmetry before generating the CFA image. This padding was removed before computing PSNRs. Extra care was taken to make sure that the CFA images used for each method were the same. We also used the parameters recommended by the authors for each method. Table 7.2 shows that not only do these methods perform well in terms of PSNR but that they are also reasonably fast. The method [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] is written in Python and the methods [TZZZ17, MBP + 09, KMTO13, KMTO14, MKTO15] are written in Matlab. All computation times were computed on an Intel Core i7-7820HQ even for CNN methods. The CNN based methods were tested without using a GPU. We used the weights provided by the authors for the both CNN methods.

Figures 7.6, 7.7, 7.8 and 7.9 present visual examples of the different demosaicking methods. In particular Figure 7.7 shows that the method with the least moiré artifacts is the method that was trained specifically to avoid this artifact. Figure 7.8 shows that both CNN based methods have no visible zipper effects.

Overall, the two reviewed CNN methods outperform both visually and in terms of PSNR all previous demosaicking methods while also being reasonably fast, even without a GPU.

Conclusion

We have seen that the CNN based demosaicking methods beat by almost two decibels the best human-crafted methods, while being faster by one order of magnitude. To reach this performance, they did not rely on the clever human techniques established by the anterior state of the art, but simply applied rather standard CNN architectures. This success is explainable. The first reason is that human-crafted algorithms rely on the iteration of nonlinear local filters, the most complex one, ARI, having more than 20 such iterations. But deep convolution networks have the same structure and are in addition scalable in depth and number of filters, until they reach the best performance. Furthermore, human-crafted methods have been designed to avoid certain artifacts, probably to the cost of losing in PSNR. CNNs, on the contrary, can be taught to learn intricate casuistry by the variety of presented of images, and to keep a memory of complex image statistics. Being trained to reach the best PSNR, they definitely reach that goal. Finally, the Gharbi et al. method succeeded in biasing the learning process so as to avoid completely the most annoying moiré and zipper effects, thus achieving the most visually pleasing results of all methods, while still retaining the best PNSR! This shows a clear superiority of deep learning based method over traditional ones and motivates the need for the self-supervised learning presented in the next chapter.

Joint demosaicking and denoising by fine-tuning of bursts of raw images

Demosaicking and denoising are the first steps of any camera image processing pipeline and are key for obtaining high quality RGB images. A promising current research trend aims at solving these two problems jointly using convolutional neural networks. Due to the unavailability of ground truth data these networks cannot be currently trained using real RAW images. Instead, they resort to simulated data. In this chapter we present a method to learn demosaicking directly from mosaicked images, without requiring ground truth RGB data. We apply this to learn joint demosaicking and denoising only from RAW images, thus enabling the use of real data. This allows to mitigate the training bias due to simulated data.

In addition we show that for this application fine-tuning a network to a specific burst improves the quality of restoration for both demosaicking and denoising. The burst processing is done by fine-tuning a pre-trained network over the pairs of the specific burst. This work has been published in [START_REF] Ehret | Joint demosaicking and denoising by fine-tuning of bursts of raw images[END_REF].

Introduction

As we have seen in Chapter 7, most camera sensors capture a single color at each photoreceptor, determined by a color filter array (CFA) located on top of the sensor. The most commonly used CFA is the so-called Bayer pattern, consisting of a regular subsampling of each color channel. This means, not only that each pixel of the resulting raw image contains one third of the necessary information, but also that the color channels are never sampled at the same positions. The problem of interpolating the missing colors is called demosaicking and is a challenging ill-posed inverse problem. To further complicate things, the captured data is contaminated with noise. For these reasons the first two steps of a camera processing pipeline are demosaicking and denoising. Traditionally, these problems have been treated separately, but this is suboptimal. Demosaicking first a noisy RAW image correlates the noise making its subsequent denoising harder [PKL + 09]. Alternatively, if denoising is applied on the mosaicked data it becomes harder to exploit the cross-color correlations, which are useful for color image denoising [DFKE07a, DVF + 09].

Until recently, just like for demosaicking in Chapter 7, state-of-the-art methods for joint demosaicking and denoising were based on carefully crafted heuristics, such as avoiding interpolation across image edges [HP06, PKL + where the heuristics are encoded as a prior model [CM12, HEK + 14]. In [WGDE + 19] both problems are addressed simultaneously by aligning and fusing RAW bursts of frames.

Recent data-driven approaches have significantly outperformed traditional model-based methods [KNJF14, GCPD16, KHKP16, KL18a, KL18b, SCC18]. In [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF], state-of-the-art results are reported with a network trained on a special dataset tailored to demosaicking in which hard cases are over-represented. In [START_REF] Kokkinos | Deep image demosaicking using a cascade of convolutional residual denoising networks[END_REF] an iterative neural network is proposed, later improved by [START_REF] Kokkinos | Iterative residual network for deep joint image demosaicking and denoising[END_REF] obtaining state-of-the-art performance on both real and synthetic datasets. These networks are relatively lightweight and do not need a lot of training data. The authors in [START_REF] Syu | Learning deep convolutional networks for demosaicing[END_REF] propose two networks for demosaicking. They train on several CFA patterns to compare performance and integrate the handling of denoising with a fine-tuning step. In [START_REF] Zhao | Loss Functions for Image Restoration With Neural Networks[END_REF] the authors find that the artifacts of challenging cases are better dealt with L 1 norm, or their proposed combination of the L 1 norm with MS-SSIM. Meanwhile in [START_REF] Kwan | Comparison of deep learning and conventional demosaicing algorithms for mastcam images[END_REF] alternative metrics to PSNR are also considered.

The major difficulty in training data-driven demosaicking and denoising methods is the difficulty to obtain realistic datasets of pairs of noisy RAW and ground truth RGB images. For this reason demosaicking networks are trained with simulated data generated by mosaicking existing RGB images. However simulated data follows a statistic that can be different from real data. The RGB images used for training have already been processed by a full ISP (Image Signal Processors) pipeline which includes demosaicking and denoising steps which leave their footprint on the output image. Additionally, the Poisson noise model is only an approximation to the real noise of a specific camera. Several factors can cause deviations. For example the noise can have spatial variations due to temperature gradients in the sensor, or caused by the vignetting or the electronic components in its surroundings.

The need for a specific treatment of realistic noise has been identified in the denoising literature. Indeed most of the existing works target synthetic types of noise, e.g. Gaussian noise. Since the noise distribution is well defined, specific methods can be crafted [START_REF] Dabov | Image denoising with block-matching and 3D filtering[END_REF][START_REF] Lebrun | A Nonlocal Bayesian Image Denoising Algorithm[END_REF][START_REF] Gu | Weighted nuclear norm minimization with application to image denoising[END_REF] and data can be simulated with ground truth so to train neural networks [ZZC + 17a,ZZZ18]. However, it has been shown recently in [START_REF] Plotz | Benchmarking denoising algorithms with real photographs[END_REF] and [START_REF] Abdelhamed | A high-quality denoising dataset for smartphone cameras[END_REF] that networks trained on synthetic noise often fail to generalize to realistic types of noise. This has started a trend of study of "real noisy images". For example [CCXK18, GYZ + 18] acquire datasets where a low-noise reference image is created by using a longer exposure time. Creating this type of dataset is time consuming and prone to bias, as to avoid motion blur in the long exposure the images need to be acquired with a tripod and the scene has to be static. Contribution In this chapter we introduce a mosaic-to-mosaic training strategy analog to the noise-to-noise [LMH + 18] and the model-blind fine-tuning presented in Chapter 6 frameworks to be able to handle mosaicked RAW data. The trained network learns to interpolate two thirds of the image data, without having ever seen a complete image. This allows us to train both demosaicking and joint demosaicking and denoising networks without requiring ground truth. The resulting networks attain state-of-the-art results, thus eliminating the need to simulate simplistic noise models or to capture time-consuming datasets with long exposure reference frames. Although we show results only with a Bayer pattern, our method can equally be applied to other CFA patterns, such as the Fujifilm X-Trans. To the best of our knowledge, this is the first method that learns joint demosaicking and denoising without any ground truth whatsoever; the network has only seen noisy mosaicked images.

With the proposed framework, we can fine-tune a pre-trained network to a RAW burst. This allows leveraging the already available multi-frame burst data that is present on many mobile camera phones [WGDE + 19]. The fine-tuning not only adapts the network to the specificities of the camera noise, but it also overfits to the burst. We demonstrate that this overfitting, when controlled, can be beneficial. A similar conclusion in the context of single-image super-resolution was reached by the authors of [START_REF] Shocher | Zero-Shot Super-Resolution Using Deep Internal Learning[END_REF]. Additionally, when used with an L 1 loss, the fine-tuned network naturally handles noise clipping, a common but challenging problem [LMH + 18,ZSC19].

The proposed strategy can be used to fine-tune other demosaicking networks, for example in this chapter we show this for the network of [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] (see Figure 8.1), but it could be used in conjunction with more recent burst denoising networks such as [START_REF] Godard | Deep burst denoising[END_REF] and [MBC + 18b] adapted for CFA images.

The rest of the chapter is organized as follows. In Section 8.2 we present the proposed mosaicto-mosaic training of a demosaicking network from a dataset of RAW mosaicked data without ground truth. In Section 8.3 we address the problem of joint demosaicking and denoising given a burst of RAW mosaicked noisy images. Results are shown in Section 8.4.

Learning demosaicking w/o ground truth

In this section, we propose a learning method to train demosaicking networks without any ground truth RGB images. Consider two different mosaicked pictures of a same scene I 1 and I 2 . We shall use one image as partial ground truth to learn demosaicking the other (provided that there is a slight movement between the two, so that with high probability the mosaic patterns do not match).

Our method requires that the two pictures can be registered, which is possible when the viewpoints are not too different. This condition is typically met for bursts of images. Modern cameras systematically take bursts of images, these sequences allow to eliminate shutter lag, to apply temporal noise reduction, and to increase the dynamic range of the device. Nevertheless, the pair of pictures can also be acquired manually by taking two separate pictures of the same scene.

In the following, we suppose we have a set of pairs of images (for example extracted from bursts), where each pair (I 1 , I 2 ) consists of pictures of the same scene for which we have estimated a transformation T that registers I 1 to I 2 . An affinity is often sufficient to accurately register a pair of images from a burst. Pairs that cannot be registered accurately won't be processed together. The original mosaicked image can be obtained from its demosaicked one by masking pixels. Thus, if we apply a demosaicking network D to I 1 , then apply the transformation T followed by the mosaicking mask, we are supposed to get I 2 . We can compute M (T (D(I 1 ))), where M represents the mosaicking operation (masking pixels), compute a distance to I 2 , which acts as ground truth, and backpropagate the gradient to train D. In some sense, I 2 acts as a partial ground truth, as only parts of T (D(I 1 )) gets compared to I 2 (for a Bayer pattern, one fourth is compared for the red and blue channels, one half for the green channel). However, contrary to artifical RGB ground truths, we do not suffer from bias introduced by the RGB processing pipeline, nor require complex settings to produce these RGB ground truths. We implemented T with a bicubic interpolation through which gradient can be backpropagated. This results in the following loss:

p (D(I 1 ), I 2 ) = M (T (D(I 1 ))) -I 2 p p , ( 8.1)
where p = 1, 2. The norm is computed only in the pixels where both images are defined. In this section we use p = 2 (squared L 2 norm). The training method is depicted in Figure 8.2. Our learning process can be linked with [LMH + 18, BR19]. The main difference is that we have an a priori on the position of the degraded pixels.

Demosaicking network To test the proposed training, we will use in this section a network architecture heavily inspired by the one from Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF], as we have seen Chapter 7 that it performs very well, while using improvements suggested in more recent work with the usage batch normalization layers [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] as well as residual learning [START_REF] He | Deep residual learning for image recognition[END_REF]. These techniques are known to speed-up training time and sometimes increase performance. The network starts with a four-channel Bayer image that goes through a series of 14 Conv+BN+ReLu layers with 64 features and 3 × 3 convolutions. A 15th layer of Conv+BN+ReLu produces 12 features with 3 × 3 convolutions. It is followed by an upsampling layer producing an RGB image of twice the width and twice the height. Like Gharbi et al. we added a layer (a Conv+BN+ReLu with 3 × 3 convolutions) before the layer producing the final output. We need add the bilinearly interpolated RGB image to the output of our network to produce the final result. All convolution layers have padding to keep the resolution constant from beginning to end. The architecture of the network is depicted in Figure 8.3. Comparing learning with ground truth RGB and our method We verify that this method for training demosaicking without ground truth is competitive with classic supervised training by training the same architecture with both methods and show comparable results. For this experiment we considered a mosaicking with Bayer pattern which is the most frequent mosaicking pattern.

In order to be able to compare the results of training with and without ground truth, we decided to simulate the pairs on which the demosaicking is trained. For both trainings we use the dataset of [START_REF] Syu | Learning deep convolutional networks for demosaicing[END_REF], which consists of 500 images (of sizes around 700 × 500) from Flickr. To generate pairs to learn with our method, we warped the same RGB image with a random affinity -thus simulating two views -and generated the mosaicked images from them. To speed-up the training we chose the same transform for all patches of a same batch. We trained both networks for 45 epochs using Adam and a learning rate of 10 -2 . We reduced the learning rate by a factor of 10 at epochs 20 and 40.

Figure 8.4 compares the evolution of the PSNR on the validation dataset (we use here the Kodak dataset1 ) while training our network with ground truth against the training without ground truth. It can be observed that training without ground truth behaves the same as with the ground truth. The convergence speed seems to be equivalent as well as the final demosaicking quality.

Table 8.1 shows the quality of demosaicking using either ground truth or no ground truth versus the state of the art in image demosaicking. The model learned without having ever seen an RGB image is able to achieve the same quality than the same network trained using the RGB ground truth, which indicates that having a ground truth is not necessary to obtain state-of-the-art performance on this task. For comparison, we also show the results obtained with model-based methods [Get11, HEK + 14] that do not need training with ground truth (they do not need training at all).

Joint demosaicking and denoising by fine-tuning on a burst

In the previous section, we demonstrated that having a training dataset with RGB ground truth is not mandatory to reach state-of-the-art performance: Similar demosaicking performance is reached with a just a database of pairs of mosaicked data. While this was demonstrated on synthetic noise-free images, it can also be done when images are actual noisy RAW data. In this section we show an application of the method to online fine-tuning on bursts. We present the method on two networks: the noiseless network from Section 8.2 where the network has to learn to denoise using only the burst (this is a sort of a toy example) and the state-of-the-art network from [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF].

Joint demosaicking and denoising without ground truth Using the noise-to-noise framework presented in [LMH + 18], we aim to train a network with parameters θ. Supervised learning of a joint demosaicking and denoising network corresponds to solving arg min

θ i L(f θ (x i ), y i ), (8.2)
where the x i are noisy mosaicked images, and the y i are their ideal noise-free demosaicked images, L is a loss such as L 2 or L 1 . In the noise-to-noise framework, the equivalent problem (conditionally on the noise being mean preserving for L 2 , or median preserving for L 1 ) is to solve arg min

θ i L(f θ (x i ), x i ), (8.3)
where x i and x i are two noisy observations of the same scene.

Our proposal is to solve arg min

θ i 1 (f θ (x i ), x i ), (8.4)
where the (x i , x i )s are pairs of noisy mosaicked images of the same scene and p is the loss introduced in Equation (8.1). We use p = 1 in this section (L 1 norm), which allows to handle clipped noise (see discussion on the choice of the loss).

The loss requires the computation of a transform T matching each pair of mosaicked images. For that we use the inverse compositional algorithm [START_REF] Thevenaz | A pyramid approach to subpixel registration based on intensity[END_REF][START_REF] Baker | Equivalence and efficiency of image alignment algorithms[END_REF] to estimate a parametric transform (in practice we estimate an affinity which we found to be well-suited for bursts). An implementation of this method is available in [START_REF] Briand | Improvements of the Inverse Compositional Algorithm for Parametric Motion Estimation[END_REF]. The advantage of this method is that it is robust to noise and can register two images very precisely (provided that they can be registered with an affinity). Since we only have access to Bayer images of size W × H, the first step is to generate four-channel images of size W 2 × H 2 corresponding to the four phases of the Bayer pattern. The transform is then estimated on these images before upscaling it to the correct size.

Having the pairs with the associated transform, one can finally apply the pipeline presented in Section 8.2 and in Figure 8.2. As in [EDM + 19] we initialize the network using a pretrained one. In particular in the following, we use the network trained for demosaicking without ground truth presented in Section 8.2, as well as the network from [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF].

Choice of Loss One particularly well known problem with denoising is clipped noise: The underlying signal I belongs to a fixed range, but the noise can make it leave that intensity range. Due grayscale images with saturated regions. From left to right: PSNRs over the whole image, on non-saturated regions, and on the saturated regions. After fine-tuning, the network works better both on saturated and non-saturated regions. While L2 is not able to deal with clipping, using L1 for fine-tuning performs similarly to fine-tuning without clipping. to hardware clipping, the measured image is inside the fixed range, and thus the noise statistics are biased. When minimizing with the L 1 norm over the same image with several noise realizations, the best estimator is the median of the realizations [LMH + 18], which is unaffected by the hardware clipping. Thus by using L 1 norm and fine tuning on a burst, our method handles clipping without any pre or post-processing required. This phenomenon is illustrated in Figure 8.5 with a classic denoising network, DnCNN [ZZC + 17a].

Fine-tuning to a single scene By fine-tuning over a single burst the network ends up overfitting the data. Usually overfitting to the training data is avoided as it results in a poor generalization ability. However, in our case the fine-tuned network will only be applied to that burst, and overfitting improves the result for that specific burst. There are other examples in the literature where a network is overfitted to a specific input (or a small dataset of inputs). For example, [CMPT + 17] turns an object classification network into a video segmentation one by fine-tuning it on the first frame, which is labeled. The network then learns to track the labeled objects in the following frames.

This fine-tuning is also reminiscent of traditional image processing methods that fit a model 158 to the patches of the image. In [START_REF] Yu | Solving inverse problems with piecewise linear estimators: From gaussian mixture models to structured sparsity[END_REF] the image patches are modeled using a Gaussian mixture model (GMM), in [START_REF] Elad | Image denoising via sparse and redundant representations over learned dictionaries[END_REF] by representing them sparsely over a learned dictionary, and in [START_REF] Mørup | Shift invariant sparse coding of image and music data[END_REF] via sparse convolutions over a set of kernels. In all these cases the models were trained on the input image. The assumption underlying these methods is that images are selfsimilar and highly redundant, allowing for compact representations of their patches.

Figure 8.6 shows that fine-tuning a grayscale denoising network (DnCNN) on a burst of images can significantly improve the denoising results. The likely explanation is that the network is able to capture a part of the image self-similarity, similar to the model-based methods. Figure 8.7 illustrates the performance evolution when fine-tuning a denoising network on a set of noisy realizations of two synthetic images, one of stripes (thus very self-similar) and a binary noise image (thus not self-similar). The performance gap is explained by the self-similarity of the former image.

Experimental results

To evaluate quantitatively the performance of the proposed training strategy, we first apply it on simulated data, since there are no real noisy raw bursts with ground truth publicly available. We generate the burst from a single image by applying random affinities. In the cases where noise is considered, the added noise is white Gaussian. During training, the affinities are estimated from the noisy raw data. Code to reproduce the results is available on https://github.com/ tehret/mosaic-to-mosaic Network fine-tuning on a sequence outperforms single image denoising Fine-tuning a network to a sequence allows to restore the image beyond the performance of a single image denoising. In this paragraph we consider a sequence of images with no motion and without mosaicking pattern. Figure 8.8 shows the PSNR evolution as the fine-tuning processes all the pairs of a sequence of 10 frames (90 pairs in total) without mosaicking pattern nor motion is considered. We consider the pairs in lexicographical order, that is every time a new input image is selected it is sequentially paired with all other images in the sequence. Note the characteristic shape traced by Table 8.2: PSNR results using an fine-tuned network on the lighthouse image of the Kodak dataset (kodim19) versus the same network without fine-tuning. While fine-tuning improves on the specific image, the overall performance on the dataset is decreased.

the PSNR curve: every time a new input image is selected the performance first drops and then steadily improves surpassing the previous peak. This shows that not only the network is adapting the current input image but it is also building upon previously seen images.

We argue that this fine tuning can be linked to a temporal noise reduction (TNR). We compare our fine-tuning based denoising strategy with three other denoising strategies: simply averaging the frames ("Average"), which amounts to a naive TNR, denoising a single frame with DnCNN ("DnCNN (25)"), which doesn't take advantage of the multiple frames, and denoising the naive TNR result with the optimal DnCNN ("Av. + DnCNN (10)"), which amounts to the best possible TNR result in this ideal case. Note that the fine tuning is largely surpassing the performance of single image denoising and approaches TNR with DnCNN. In practice temporal averaging followed by denoising is not as straightforward on mosaicked images, so there is no equivalent of this upper bound on mosaicked images. This justifies the relevance of the proposed method.

Improving demosaicking by fine-tuning Similarly to denoising, fine-tuning improves demosaicking. The evolution of the improvement, showed in Figure 8.10, is quite similar to the one presented for denoising. Moreover artifacts that existed in the initial network, due to a low amount of training, are removed completely by the fine-tuning, see Figure 8.9. The result then looks visually very similar to the result from Gharbi et al.that was trained specially to deal with these difficult cases.

Table 8.2 compares the PSNR obtained for different networks on the Kodak dataset. The network from Section 8.2 was fine-tuned on kodim19, which is singled out in the table. As expected, the fine-tuned network works well on the reference image but its performance decreases on the other images. The network without fine-tuning performs better on the whole Kodak dataset than the network that was fine-tuned on a specific image. The increase in performance for this reference image after fine-tuning was of more than 2dB. It shows that even though our method is completely blind, it is able to compete with the state of the art. The rows identify different images from the Kodak dataset, and noise levels. Moreover increasing the length of the burst also allows to improve the quality in the cases where it might perform worse otherwise. Our method used the network trained in Section 8.2 and was fine-tuned with 10 generated noisy images except when mentioned otherwise.

Joint demosaicking and denoising using fine-tuning The final application of fine-tuning is to do both previous applications at the same time. Table 8.3 compares our method to two other methods of joint demosaicking and denoising. The networks were fine-tuned on each image individually. Overall our fine-tuning approach is very competitive. A network that has not been trained with noisy data prior to fine-tuning ( §8.2 + our fine-tuning) is now able to perform at the same level as one of the best network trained for this specific application. When using the state-of-the-art network from [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF], our fine-tuning improves the final quality by more than 1dB.

Not only do we achieve competitive results in terms of PSNR, the results are free of demosaicking artifacts. Indeed, as shown in Figure 8.1, even in the regions that are particularly hard such as the fence. For example there is no zippering compared to the result of Gharbi et al..

The final experiment is on real data. We took a burst from the HDR+ dataset [HSG + 16] and applied our process. We compare the result of a simple bilinear interpolation, the result of [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] and [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] with our fine-tuning in 8.11. Fine-tuning allows for a better denoising + 16], bilinear interpolation, [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] and [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] with our fine-tuning. Contrast was enhanced for all methods except HDR+. Note that the HDR+ pipeline includes color balance as well as sharpening. It also uses all the images of the burst to produce the result (all other methods use only the reference frame). Figure best visualized zoomed in on a computer. and a better reconstruction of details while limiting artifacts. We present additional result on the HDR+ dataset in Figures 8.12, 8.13, 8.14. In these figures we show the result from the HDR+ pipeline [HSG + 16], a bilinear interpolation, the result of Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF], and the result of Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] with our fine-tuning.

Remarks on computation cost

We empirically found that the amount of data needed for finetuning the network is linked to the number of pixels and not the number of images of a burst. This allows to fine-tune even on short bursts like the ones from the HDR+ dataset (of size 2400×1300) using at most six images. Regarding computation time, we presented fine-tuning as an offline application, for example for professional photography where best quality is required. However, recent works [TRJ + 19,TTP + 19] have shown that fine-tuning can also be achieved in real time for videos.

Conclusion

In this chapter, we have proposed a novel way of training demosaicking neural network without any RGB ground truth, by using instead other mosaicked data of the same scene (such as from a burst of images). Based on it and on recent neural network advances, we proposed a method to train jointly demosaicking and denoising with bursts of noisy raw images. We show that finetuning on a given burst both boosts the reconstruction performance and handles clipped noise natively. It also presents a specific case where overfitting a network to the training data is valuable. Since we do not expect generalization there's only benefits from this overfitting.

We hope our work can lead to new camera pipeline calibration procedures, and general improvement of the image quality when a burst is available.

How to Reduce Anomaly Detection in Images to Anomaly Detection in Noise

Anomaly detectors address the difficult problem of detecting automatically exceptions in a background image. The background image can be as diverse as representing fabric or a mammography. Detection methods have been proposed by the thousands because each problem requires a different background model. By analyzing the existing approaches, we show that the problem can be reduced to detecting anomalies in residual images (extracted from the target image) in which noise and anomalies prevail. Hence, the general and impossible background modeling problem is replaced by a simple noise model, and allows the calculation of rigorous detection thresholds. Our approach is therefore unsupervised and works on arbitrary images. The residual images can favorably be computed on dense features of neural networks. Our detector is powered by the a contrario detection theory, which avoids overdetection by fixing detection thresholds taking into account the multiple tests. This work was published in [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] and [START_REF] Ehret | Image anomalies: A review and synthesis of detection methods[END_REF].

Introduction

The automatic detection of anomalous structure in arbitrary images is concerned with the problem of delineating image regions not conforming with the rest of the image. This is a challenging computer vision problem, as there seems to be no straightforward definition of what is (ab)normal for a given image. Anomalous structure in images can be generally described as being either caused by high-level or low-level outliers. High-level anomalies are related to the semantic information presented in the scene. For example, human observers immediately detect a person inappropriately dressed for a given social event. In this chapter, we focus on the problem of detecting anomalies due to lowor mid-level rare events (e.g., patterns) present in images. This is an important problem in many industrial or biomedical applications where a fast and reliable way of detecting rare patterns is needed.

We propose here an unsupervised method for detecting anomalies in an arbitrary image. The method doesn't rely on a training dataset of normal or abnormal images, neither on any other prior knowledge about the image statistics. It directly detects anomalies with respect to residual images estimated solely from the image itself. We only use a generic, qualitative background image model: we assume that anything that repeats in an image is not an anomaly. Hence we extract from the input image a residual image containing everything that does not repeat. This unstructured image is akin to noise, but still contains the anomalies if any.

Detecting anomalies in noise is far easier and can be made rigorous and unsupervised by the a-contrario theory [START_REF] Desolneux | From gestalt theory to image analysis: a probabilistic approach[END_REF] which is a probabilistic formalization of the non-accidentalness principle [START_REF] Lowe | Perceptual organization and visual recognition[END_REF]. The a-contrario framework has produced impressive results in many different detection or estimation computer vision tasks, such as, segment detection [START_REF] Grompone | Lsd: A fast line segment detector with a false detection control[END_REF], ellipse detection [START_REF] V Patraucean | A parameterless ellipse and line segment detector with enhanced ellipse fitting[END_REF], spots detection [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF], vanishing points detection [START_REF] Lezama | Finding vanishing points via point alignments in image primal and dual domains[END_REF], fundamental matrix estimation [START_REF] Moisan | A probabilistic criterion to detect rigid point matches between two images and estimate the fundamental matrix[END_REF], mirror-symmetry detection [START_REF] Viorica Patraucean | Detection of mirror-symmetric image patches[END_REF], among others. The fundamental property of the a-contrario theory is that it provides a way for automatically computing detection thresholds that yield a control on the number of false alarms (NFA). It follows that not only one can detect anomalies in arbitrary images without complex modeling, but in addition the anomalies are associated an NFA which is often very small and therefore offers a strong guarantee of detection.

In a nutshell, our method removes from the image its self-similar content (considered as being normal). The residual can be modeled after a simple equalization as Gaussian noise, but still contains the anomalies according to their definition: they do not repeat. We shall show detections performed directly on the residual, or alternatively on residuals extracted from dense low-and mid-level features of VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] the popular pre-trained deep neural network. Our method is general and works equally well on these different image representations.

A preliminary short version of this work was published in a conference [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF]. The anomaly detection method developed here is also described briefly in our review paper [START_REF] Ehret | Image anomalies: A review and synthesis of detection methods[END_REF]. Our Section 2 below summarized some of the conclusions about the literature contained in this last paper. The present version incorporates a more detailed analysis of the detection method and its implementation.

The remainder of the chapter is organized as follows. Section 9.2 discusses in detail previous work and the substantial differences to what we propose. Section 9.3 explains the method and its implementation. In Section 9.4 we present results of the proposed method on both real and synthetic data, comparing the algorithm to other state-of-the-art anomaly detection methods. We finally close in Section 9.5. .2: Our method applies the same framework at all the different scales. The framework computes the difference between the image (possibly using a dense representation using a neural network) and its self-similar model before normalizing this difference output distribution. The detection is then done on these normalized distributions. The corresponding pseudocode is presented in 20.

An analysis of the literature

The 2009 review [START_REF] Chandola | Anomaly detection: A survey[END_REF] examining some 400 papers on anomaly detection considered allegedly all existing techniques and all application fields. It is fairly well completed by the more recent [START_REF] Marco | A review of novelty detection[END_REF] review. These reviews agree that classification techniques like SVM can be discarded, because anomalies are generally not observed in sufficient number and lack statistical coherence. There are exceptions like the recent method [START_REF] Ding | An experimental evaluation of novelty detection methods[END_REF] which defines anomalies as exceptional events that cannot be learned, but after estimating a background density model, the right detection thresholds are nevertheless learned from anomalies. A broad-related literature exists on saliency measures, for which learning from average fixation maps by humans is possible.

For example [START_REF] Hamed Rezazadegan Tavakoli | Fast and efficient saliency detection using sparse sampling and kernel density estimation[END_REF] trained on average human fixation maps to learn both the anomalies and their surround vectors as Gaussian vectors. This reduces the problem to a two class Bayesian classification problem. The goal of saliency detectors is only to deliver a fuzzy saliency map. Anomaly detectors instead signal anomalous regions. The saliency detectors try to mimic the human visual perception and in general introduce semantic prior knowledge related to the perceptual system (e.g., face detectors). This approach works particularly well with neural networks because attention maps obtained by gaze trackers can be used as a ground truth for the training step. SALICON [START_REF] Huang | Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks[END_REF] from Huang et al.is one of these deep neural networks architecture achieving state of the art performance. In [START_REF] Kumar | Neural network based detection of local textile defects[END_REF] a neural network is trained on a base of defect/non-defect, thus again performing two classes classification. But the anomaly detection problem has been generally handled as a "one class" learning problem. The 2003 very complete review by Markou and Singh [START_REF] Markou | Novelty detection: a review -part 1: statistical approaches[END_REF] concluded that most research on anomaly detection was driven by modeling background data distributions, to estimate the probability that test data do not belong to such distributions. Hence the mainstream methods can be classified by their approach to background modeling.

Probabilistic background models

Their principle is that anomalies occur in the low probability regions of the background model. This stochastic model can be parametric (Gaussian, Gaussian mixture, regression), or nonparametric. For example in "spectral anomaly detection", an anomaly is defined by having deviant coordinates with respect to normal PCA coordinates. In [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF] a Gaussian background Fourier model of the image phase is followed by a Mahalanobis threshold. In [START_REF] Du | Random-selection-based anomaly detector for hyperspectral imagery[END_REF] a Gaussian background model from random pixels is similarly followed by a Mahalanobis threshold. In [START_REF] Goldman | Anomaly detection based on an iterative local statistics approach[END_REF] the background is characterized in a feature space of principal components, and hypothesis testing is used for the detection of anomalous pixels. In [START_REF] Tarassenko | Novelty detection for the identification of masses in mammograms[END_REF] the assumption is made that abnormalities are uniformly distributed outside the boundaries of normality, defined as the probability density estimation of the training data. In [START_REF] Honda | Finding "anomalies" in an arbitrary image[END_REF] Honda and Nayar introduced a generic method which works on all type of images. The main idea is to estimate the probability of a region conditioned on the surroundings. The method employs independent component analysis to find a compact representation of the region space and its surroundings. The [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF] method models the background image as a Gaussian stationary process. This is rather restrictive, but this precise model allows computing an accurate Number of False Alarms [START_REF] Desolneux | From gestalt theory to image analysis: a probabilistic approach[END_REF] for anomalies. Thus, in methods relying on a probabilistic background model, outliers are detected as incoherent with a probability distribution estimated from the input image(s). The anomaly threshold is a statistical likelihood test on the learned background model. We now pass to non-stochastic background models.

NN-based background reconstruction. "Replicator" neural networks [START_REF] Hawkins | Outlier detection using replicator neural networks[END_REF] can model a background. These are multi-layer feed forward neural networks with same number of input and output neurons. The training involves compressing data into hidden layers. The testing phase reconstructs each data sample. Its reconstruction error for the test instance is used as an anomaly score. The work in [SSW + 17] is also equivalent to using an autoencoder and looking at the norm between the original and the output. A GAN is trained (generator + discriminator) via gradient descent, a representation in latent space is computed, and the output is compared to the input. The discriminator cost is then used alongside the representation on the input by the network to find the anomalies. This paper is related to [START_REF] An | Variational Autoencoder based Anomaly Detection using Reconstruction Probability[END_REF] that computes a reconstruction probability from a variational autoencoder.

Fourier Background subtraction. Perhaps the most successful background based method is the detection of anomalies in periodic patterns of textile [TH99, TH03, PCC10]. This can be done naturally by cutting specific frequencies in the Fourier domain and thresholding the residual to find the defects.

Non constructive background models

These methods present the big advantage that they no longer require the construction of a background model, which for most images is anyway impossible. Hence, they simply make structural assumptions on the background image that would be violated by anomalies.

Center-surround enhancement. These methods are mainly used for creating saliency maps. Their rationale is that anomalies pop up as local events contrasted with their surroundings. In [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF], center surround detectors based on color, orientation and intensity filters are combined to produce a final saliency map. Detection is then done on a simple winner-takes-all scheme on the maximum of the response maps. In [START_REF] Murray | Saliency estimation using a non-parametric low-level vision model[END_REF] a saliency map is obtained from center-surround contrast coefficients for wavelet filters. An image wavelet pyramid reconstruction with these coefficients enhances local anomalies. Detection in images and videos is also done in [START_REF] Gao | The discriminant centersurround hypothesis for bottom-up saliency[END_REF] with center-surround saliency detectors which stem from [START_REF] Itti | A saliency-based search mechanism for overt and covert shifts of visual attention[END_REF] adopting similar image features. In [START_REF] Honda | Finding "anomalies" in an arbitrary image[END_REF], the main idea of a fast and general anomaly detection method is to estimate the probability of a region conditioned on the surroundings. The method employs ICA and KLT to find a compact (with elements as independent as possible) of the region space and its surroundings.

The sparsity model. A more recent nonparametric trend is to learn a sparse dictionary representing the background (i.e., normality) and to characterize outliers by their non-sparsity. In [START_REF] Margolin | What makes a patch distinct?[END_REF] the patch background model is simply its PCA and the patch saliency is computed as the L 1 norm of the patch coefficients in PCA. Aggregating these values gives a pixel saliency. The saliency formation in [START_REF] Xie | Texems: Texture exemplars for defect detection on random textured surfaces[END_REF] builds a Gaussian mixture model for patches, and probability thresholds are learned on images without anomalies. The final result is a saliency map. In [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] the background model is a learned patch dictionary from a database of anomaly-free data. The abnormality of a patch is measured as its Mahalanobis distance to a 2D Gaussian learned on the parameter pairs composed by the L 1 norm of the coefficients and of their reconstruction error. An extension [CBFW15] learns a convolutional sparse dictionary. Similarly in [START_REF] Elhamifar | See all by looking at a few: Sparse modeling for finding representative objects[END_REF] a patch is anomalous when the L 1 norm of its sparse decomposition on a learned dictionary is too large.

Defining anomaly detection as a variational problem where anomalies are detected as nonsparse is also the core of the method proposed in [START_REF] Adler | Sparse coding with anomaly detection[END_REF]. The L 1 norm of the coefficients on the learned background dictionary is used as an anomaly measure.

The self-similarity model. The self-similarity principle has been successfully used in many different applications. In particular in image denoising such as the bilateral filter [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF] or nonlocal means [START_REF] Buades | A non-local algorithm for image denoising[END_REF] for example; but also for texture synthesis in the pioneering work by Efros and Leung [START_REF] Efros | Texture synthesis by non-parametric sampling[END_REF]. The basic assumption of this generic background model, applicable to most images, is that in normal data, features are densely clustered. Anomalies instead occur far from their closest neighbors. This idea can be implemented by clustering (anomalies being detected as far away from the centroid of their own cluster), or by nearest neighbor search (NN). NN search leads to simple direct rarity measurements. For example in [SM09] the saliency measure

is S i = N j=1 exp -1+ρ(F i ,F j ) σ 2 -1
where F i are local features, and F j the closest features to F i . If all F j are far away from F i , the saliency is high. The algorithm in [START_REF] Zontak | Defect detection in patterned wafers using anisotropic kernels[END_REF] is inspired from NL-means [BCM05a]: a) fix a similarity threshold learned in a reference image without anomalies and b) compare each patch of the source image to the patches of the reference; if the distance is higher than the similarity threshold, then the patch is an anomaly. A similar idea can be found back in [START_REF] David | Outlier detection using classifier instability[END_REF] : "The distance of the new object and its nearest neighbor in the training set is found and the distance of this nearest neighbor and its nearest neighbor in the training set is also found. The quotient between the first and the second distance is taken as indication of the novelty of the object." The self-similarity measurement in [START_REF] Stas Goferman | Context-aware saliency detection[END_REF] finds for each patch p i its 64 most similar patches q k in a spatial neighborhood and computes its saliency as S i = 1 -exp -1 64 64 k=1 d(p i , q k ) . A similar method and saliency detector is proposed in [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF]. This method performs a dimension reduction thanks to a low dimensional embedding of a nearest neighbor graph using the coordinates of points on the eigenvectors of the graph Laplacian. The anomaly score is then given by the distance to the first nearest neighbors with the diffusion distances. In [START_REF] Boiman | Detecting irregularities in images and in video[END_REF], image regions are matched (with deformation allowed) to others in the same image or video. The probability of the deformation is estimated and gives a saliency map.

Boracchi and Roveri [START_REF] Boracchi | Exploiting self-similarity for change detection[END_REF] proposed to detect structural changes in time-series by exploiting the self-similarity. Their general idea is that a normal patch should have at least one very similar patch along the sequence. Given a temporal patch (a small temporal window) the residual with respect to the most similar patch in the sequence is computed. This leads to a new residual sequence (i.e., change indicator sequence). The final step is to apply a traditional change detector test (CDT) on the residual sequence. CDTs are statistical tests to detect structural changes in sequences, that is, when the monitored data no longer conform to the independent and identical distributed initial model. CDTs run in an online and sequential fashion.

Global rarity measurement. Generally histogram based, these methods assign an anomaly score to each tested feature based on the inverse of the height of the bin to which it belongs.

Similarly in [RMD + 13] a saliency map is obtained by summing up the rarity of 32 multiscale oriented features, computed for each pixel as a weight inversely proportional to its rarity in the wavelet histogram. Patches are represented in [START_REF] Borji | Exploiting local and global patch rarities for saliency detection[END_REF] by their coefficients on a patch dictionary learned on natural images. The final saliency is the inverse of a patch probability of happening. The method is finally combined with a local center-surround saliency measure.

Discussion and our proposition

We now discuss briefly the previous classification. Background probabilistic modeling is very powerful when images belong to a restricted class of homogeneous objects, like textiles. Indeed, it furnishes rigorous detection thresholds based on the estimated probability density function. But, regrettably, this method is nearly impossible to apply on generic images. For the same reason, background reconstruction models based on CNNs are restrictive and do not rely on provable detection thresholds. Center-surround contrast methods are successful for saliency enhancement, but lack a detection mechanism. Hence, they only furnish a saliency image, not a binary anomaly decision. The sparsity and the self-similarity models are tempting and thriving. Their big advantage is their universality: they can be applied to most images. But again, excepting [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF], they lack a rigorous detection mechanism, because they work on a feature space that is not easily modeled.

Our proposition is to benefit of the advances of the above methods while avoiding their mentioned limitations. To this aim, we construct a probabilistic background model on a new feature image that we call the residual. This residual is obtained by reconstructing a self-similar version of the target image. The difference between the target and its self-similar version is called the residual and becomes our new background. Being not self-similar, this background is akin to a colored noise. Hence hypothesis testing can be applied to it, and more precisely multiple hypothesis testing (also called a contrario method), as proposed in [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF].

In that way, a general and simple method can be built that works on all images and detects anomalies by a rigorous threshold. It does not require learning, and it is easily made multiscale. Our underlying model can therefore be considered as fully generic in the sense that it decomposes any image into a self-similar part and its "residual", which contains only noise and the anomalies.

Method

The anomaly detection method that we propose is therefore built on two main blocks: a removal of the self-similar part of the image, and a simple statistical detection test based on the a contrario framework on the residual. The pipeline is summarized in Figure 9.2 and in Algorithm 20.

Self-similarity Background Modeling

The proposed self-similarity based background subtraction is inspired from patch-based denoising algorithms, but with a crucial difference. All self-similarity based denoising algorithms share a similar procedure. First, a set of similar patches is computed. This search is generally performed locally around each patch [START_REF] Dabov | Image denoising by sparse 3-d transform-domain collaborative filtering[END_REF][START_REF] Buades | A non-local algorithm for image denoising[END_REF] to keep computational cost low and to avoid noise overfitting. An "self-similar estimate" of the query patch is then built using the nearest neighbors found at the previous step. These nearest neighbors will be picked with a different rule for anomaly detection than for denoising, as we shall see next.

Self-similarity from exclusively non-local patches. Image denoising tries to maximize the quality of the result and must therefore perform acceptably even with the non self-similar parts of Add l i to l the image. For anomaly detection instead, the algorithm should perform well only if a clear structure emerges. Furthermore, we want our self-similarity search to be made on the whole image. But the main difference with classical denoising is that we forbid local comparisons. Thus, contrary to classic denoising algorithms, the search is performed outside a square region surrounding each query patch only. Otherwise any anomaly with some internal structure might be considered a structure. What matters is that the event represented by the anomaly is unique in the image.

Searching for similar patches. In image patch-based methods, searching for similar patches is usually done in a small squared region surrounding the query patch. For denoising, searching in a small region acts as a regularizer and speeds up the processing. In our case this can't be done because we forbid a local squared region around the query patch. We'd also like to take advantage of the whole image; repetitions could appear anywhere in the image and not necessarily locally. This is why we used the global search presented in Chapter 2. The search is summarized in Algorithm 23 with each step shown in Figure 9.3. Another advantage of this search is that the query image and the reference image can be different without any loss in performance. This is similar to an external search in a database made of one or multiple images (See Figure 9.4).

In this chapter we suggest using VP-trees for the global search since it has been shown to work well for a similar problem [START_REF] Ehret | Global patch search boosts video denoising[END_REF]. However, other global patch search methods, such as PatchMatch [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF], can also be used. While PatchMatch has more convergence guarantees, see Chapter 10, it is also less computation efficient. PatchMatch is very efficient when looking for very few (namely one or two) nearest neighbors but struggles for a large number of nearest neighbors. PatchMatch also requires the computation of similar patches for the entire database while the current approach only does it for the image that is being processed. Moreover this approach is easier to parallelize than PatchMatch. 5 Compute the median distance p from s to the other elements of S 6 Construct T 1 using the elements of S which are closer than p from s 7 Construct T 2 using the elements of S which are further than p from s 8 return ((s,p), T 1 , T 2 ) Background model computation. The background removal step follows the steps of the Nonlocal means algorithm [START_REF] Buades | A non-local algorithm for image denoising[END_REF]. For a similarity parameter and for each patch P in the image the n most similar patches denoted by P i are searched and averaged to get a self-similar estimate P for the patch,

P = 1 n i=1 exp - P -P i 2 2 h 2 n i=1 exp - P -P i 2 2 h 2 P i . (9.1)
Aggregating these estimated patches produce a self-similar model of the tested image. The residual is then the difference between this self-similar model and the input image. As it can be seen in Figure 9.1, this process separates the content of the image (background) from the noise and anomalies. We argue this residual is much easier to model since it doesn't contain any complex structure. Algorithm 24 gives a generic pseudocode for this process.

Residual distribution. The detection process, as presented in 9.3.2, relies on the residual being Gaussian. However, the distribution of r(u) is not necessarily Gaussian. In [ZLZ + 17] it is hinted that r(u) might follow a Laplace distribution for natural images for some denoising algorithm. The fact that non-local means transforms white Gaussian noise into white Gaussian noise is substantiated in [START_REF] Buades | Nonlocal image and movie denoising[END_REF]. Based on these remarks, we add an additional step to transform the residual (which can be either Laplacian or Gaussian) into the required distribution: We fit a few distributions on the residual (See Algorithm 7). The distribution depends on the image and on the choice of the input features, depending on whether we work directly on an image or on features from a neural network (as mentioned in Section 9.3.3). In the case of a neural network, Zeros are ignored when estimating the distribution. Indeed, when using CNNs, RELUs cause zero to be Algorithm 25: f itResidual: Fit the residual distribution and convert it to a Gaussian input : r a set of residual values obtained from Alg. 24 output : r the resulting modified residual 1 Ignore all zeros for the distribution estimation.

∀ > 0, E[#{i, f (i, X i ) ≤ }] ≤ . ( 9.2)
A common way to build an NFA is to take, for each feature channel image,

f (i, x) = N P(X i ≥ x i ), or f (i, x) = N P(|X i | ≥ |x i |) (9.3)
where X i denotes a residual image. N is the number of tests, generally the overall number of pixels of all tested images.

Detection of anomalies of different sizes at a given scale. When working with colored noise, Grosjean and Moisan [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF] recommend to convolve the noise with a measure kernel to detect spots of a certain size. This corresponds to the generation of new channels r(u) = r(u) * K where K is the normalized support of a disk of a given radius. But since we apply the detection at all dyadic scales, the tested radii are limited to a small set of N conv values (1 to 3) at each scale. The disks are normalized to keep unit variance. Because the residual is assumed to be a stationary Gaussian field, the results after the filtering are also Gaussian. In that case, the test function is then

f (r(u)) = N 1 2 erfc |r(u)| σ (9.4)
with sigma the variance of r(u). The combination of the detections on the different versions of the residual is explicitly handled by the statistical test. Thus, the inputs to the detection phase are multichannel images of different scales, where each one is assumed to be a centered Gaussian field having unit variance at each pixel.

Multiscaling. The problem of anomaly detection is fundamentally multiscale. One would like to find both anomalies at a fine scale, for example small anomalies inside a texture, or at a coarser scale, for example a hole inside a textile. If one wants to be thorough, all these anomalies need to be detected whatever the scale they appear in. Even though the detection algorithm presented previously is not inherently multiscale, it can become so by applying it to all the different scales of the image. The idea to generate the different scales, process them and combine the result is inspired by multiscale denoising such as [START_REF] Pierazzo | Multi-Scale DCT Denoising[END_REF]. The scaled images can be computed by Gaussian subsampling. A Gaussian blur of parameter λ is applied to the image before a subsampling of 2 in each direction (so 4 in total). The standard deviation of the Gaussian λ is chosen so that the image has the same blur as the original one after subsampling. This blur is assumed to be of the order of 0.8 for natural well sampled images [START_REF] David G Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. This means that the target blur before subsampling is λ = 2 × 0.8. We are searching for the blur λ such that the the Gaussian convolutions G verify

G λ = G λ (G 0.8 ) (9.5)
Since the variances of convolved Gaussians add up, this amounts to imposing a blur such that λ 2 = 0.8 2 + λ 2 (9.6)

i.e. 0.8 2 + λ 2 = (2 × 0.8) 2 (9.7)

and therefore λ = √ 3 × 0.8 ≈ 1.39.

(9.8)

The process can then be iterated to compute all the dyadic scales. The corresponding pseudocode is in Algorithm 26.

Algorithm 26: multiscaleDecomposition: decompose an image in its different scales input : An image u output : A set (u i ) of images representing the different scales 1 u 0 ← u // The first scale corresponds to the original image 2 for i = 1 to 3 do

3 u i ← G 1.39 * u i-1 //
G 1.39 is a Gaussian kernel of standard deviation 1.39, the convolution is computed using a discrete filter of size 2 4σ + 0.5 + 1 4 u i ← subsample(u i ) // Subsample by a factor 4 (2 in each direction)

The statistical test. To detect anomalies for both sides of the tails, we use the NFA given in (9.3). It remains to compute the number of tests N which is significantly larger than the number of image pixels. Indeed, the detection occurs on N scales different scales of the residuals, computed by Gaussian subsampling. The first scale is of the size of the original image, while each other scale is reduced by a factor of two. The produced residuals are of the size of their input images (in the case of the pixel method), or smaller (in the case of CNN features. See Section 9.3.3). Furthermore, the images have several channels and each test is replicated on all channels. Denoting by Ω s the set of pixels for the residual at scale s, N conv the number of convolution filters and N chan the number of channels, the number of tests is

N = N conv N chan N scales -1 i=0 |Ω s |.
(9.9)

In our case, N c han corresponds to the number of channels of the image, N conv to the number of radii tested and Add the position of the pixel and R i to l

Choice of the image features

We now describe the image features, or channels, on which anomalies are detected. This is often considered an important step in anomaly detection. We found that our detection in the residual, which is unstructured noise, is fairly independent of the channel choice. We used with equal success the raw RGB colors as channels, or the intermediate feature channels of a pre-trained neural network. To that last purpose we used the VGG network [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], a CNN trained on the ImageNet database [DDS + 09].

We removed the network padding to guarantee spatial invariance. Indeed if the network padding is kept, the feature distributions differ at the borders of the feature maps [START_REF] Raad | A survey of exemplar-based texture synthesis[END_REF]. We took the normalized version of the network and found slightly better results by working on the square root of the raw network features. Before the residual computation on NN features, we reduce their dimensionality with a PCA filter trained independently for each input image. This reduction is a compromise between the expressive power of the complete set of NN features and a compact image representation where visually almost-similar objects have similar representations. This transformation is represented on Figure 9.2.

Algorithm parameters

The main parameter of the statistical test is the number of allowed false alarms. In all of our experiments it was set to 1/100. Hence an anomaly is detected at pixel x in channel i if and only if the NFA function f (i, x) is below = 10 -2 . This means a theoretical expectation of less than 10 -2 "casual" detection per image under the null hypothesis. We shall anyway observe much smaller NFAs for the real anomalies.

Another setting is the size of the disks for the convolutions. We got better results with disks of radius one and two for the basic method working on pixels, and radius one, two and three with Neural Network features (See section 9.3.3). Neural Networks need bigger disks to remedy to the fact that medium-sized spots for the residual at a given scale may disappear at the following scale because Neural Networks features tend to ignore very small elements.

The other parameters are fixed as follows. We set the number of scales N scale to 4 in all of our tests. The patch size for Algorithm 24 is set to 8 × 8 × 3 for the basic version, while when using Neural Network features, we take the first five components with PCA and use a patch size of 5 × 5 × 5. In both cases, the number n of patches for the search is set to 16. The similarity parameter h is set to 10.

Experiments

In this section we shall compare six methods methods:

• The [GM09] stochastic parametric background model.

We denote this method by Grosjean.

• The [AT10] Fourier homogeneous model. We denote this method by Aiger.

• The [START_REF] Zontak | Defect detection in patterned wafers using anisotropic kernels[END_REF] non-local self-similar model. We denote this method by Zontak.

• The sparsity-based background model of [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF]. We denote this method by Boracchi.

• The non-local self-similar model of [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF]. We denote this method by Mishne.

• Our non-local self-similar model [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF]. We denote this method by Davy.

We use the NFA modified versions of these methods, presented in [START_REF] Ehret | Image anomalies: A review and synthesis of detection methods[END_REF], whenever it is possible. The advantage is that the detection process is then only controlled by a single parameter making them easier to compare, especially for AUC estimation.

We propose two types of experimental comparison.

• The first comparison is a qualitative sanity check. For this qualitative analysis we tested on synthetic examples having obvious anomalies of different types (color, shape, cluster), or inexistent (white noise). These toy examples provide a sanity check since one would expect all algorithms to perform perfectly on them. We will also examine the results of the competitors on challenging examples taken from anomaly detection articles.

• The second protocol is a quantitative evaluation. We generated anomaly-free images as samples of colored random Gaussian noise. Being a spatially homogeneous random process, such images should remain neutral for an anomaly detector. We then introduced small anomalies to these images and evaluated whether these synthetic anomalies were detected by the competitors. This leads to evaluate a true positive detection rate (TP) for each method on these images. We also evaluated how much of the anomaly free background was wrongly detected, namely the false positive detection rate (FP). Disposing of TP-FP pairs yields ROC curves that will be opportunely discussed. Undoubtedly, the colored Gaussian noise used in this experiment could be replaced by any other spatially homogeneous random process. We varied the background texture by varying strongly the process's power spectrum.

Qualitative evaluation

The toy examples are probably the easiest to analyze. We show the results in Figure 9.9. We generated images in the classic form used in anomaly detection benchmarks like in [RMD + 13], where the anomaly is the shape or the color that is unique in the figure. In the third toy example most rectangles are well spaced except in a small region. The anomaly therefore is a change in spatial density. Even though these examples are extremely simple to analyze, they appear to challenge several methods, as can be seen in Figure 9.9. Only [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] is able to detect accurately the anomaly in all three examples. This is explained in the second row where the residual after background subtraction is shown. In the residual details of the anomalies stand out on a noise-like background. While [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF] works well with the color and the shape, it fails to detect the spatial density anomaly. [START_REF] Zontak | Defect detection in patterned wafers using anisotropic kernels[END_REF] detects well but also lots of false detection. The other methods [ZC10, MC14, GM09] and [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] over-detect the contours of the non anomalous shapes, thus leading to many false positives. We also tried a sanity check with a pure white Gaussian noise image. This is done in the last two examples of Figure 9.9. [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF], and [GM09] soundly detect no anomaly in white noise, as expected. However a few detections are made by [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] and almost everything is detected by [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF]. It can be noted that the background model of the first three papers is directly respected in the case of white Gaussian noise, which explains the perfect result. (In the case of the model of [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF], it has to be noted that non-local means asymptotically transforms white Gaussian noise into white Gaussian noise [START_REF] Buades | Nonlocal image and movie denoising[END_REF]). The over-detection in [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] can be explained by the lack of an automatic statistical threshold. The few spurious detections in [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] show that the feature used for the detection doesn't follow a Gaussian distribution, contrarily to the method's testing assumption. It is also clear that one cannot build a sound sparse dictionary for white noise.

The same test was done after adding a small anomalous spot to the noise, and the conclusion is similar: [DEMD18,GM09] perform well, [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] has a couple of false detections and doesn't detect the anomaly. One method, [START_REF] Zontak | Defect detection in patterned wafers using anisotropic kernels[END_REF], doesn't detect anything. Finally [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] over-detects. Both noise images were taken from [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF].

We then analyze three examples coming from previous papers. The first one (first column in Figure 9.10) is a radar image of an undersea mine borrowed from [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF]. The mine is detected by [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF][START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF] without any false detections. Both [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF][START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] have false detections; [START_REF] Zontak | Defect detection in patterned wafers using anisotropic kernels[END_REF] over-detects and [AT10] misses the mine. The second example (second column in Figure 9.9) shows an example of near-periodic texture. This is one of the examples where Fourier based methods are ideally well suited. It was therefore important to check if more generic methods were still able to detect the anomaly. Two methods [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF] and [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF] fail to detect the anomaly, the other three methods performing really well. This makes the case for self-similarity and sparsity based methods, that generalize nicely the background's periodicity assumption. The final example (third column from Figure 9.10) is a real example of medical imaging borrowed from [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF] where the goal is to detect the tumor (the large white region). [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF][START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] fail to detect the tumor. A strong detection is given by [START_REF] Zontak | Defect detection in patterned wafers using anisotropic kernels[END_REF][START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] but the false alarms are also strong and numerous. Finally [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] has stronger tumor detections than [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF]) (a NFA of 10 -6.6 against 10 -2.8 ) but it has several false alarms as well.

Finally we tested the methods on real photographs taken from the Toronto dataset [START_REF] Bruce | Saliency based on information maximization[END_REF]. This clearly takes several of the methods out of their specific context and type of images (tumors in X-ray images, mine detection in sonar scans, clot detection in microfibers, wafer defects,...) On the other hand, the principles of the algorithms are general. So by testing on these examples, our goal is to explore the limits of several detection principles, not to compare these specific algorithms. Clearly some of the methods are more adapted for spatially homogeneous background than to an outdoor cluttered scene.

Another issue when using real photographs is that anomalies detected by humans may be semantic. None of the methods we consider was made to detect semantic anomalies, that can only be learned on human annotated images. Nevertheless, the tests' results are still enlightening. Detections are very different from one method to the other. The fourth example in Figure 9.10 shows a man walking in front of some trees. [AT10, GM09] and [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] don't detect anything. Both [START_REF] Zontak | Defect detection in patterned wafers using anisotropic kernels[END_REF][START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] detect mostly the trees and the transition between the road and the sidewalk. Surprisingly [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] only detects the man. Indeed in the noise like residual one can check that the man stands out. The second example shows a garage door as well as a brick wall. This time the algorithms tend to agree more. The conspicuous sign on the door is well detected by all methods as well the lens flare. A gap at the bottom between the brick wall and the door is detected by [DEMD18, MC14, GM09, BCW14]. The methods [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] and [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] also detect the transition between the wall and and the brick wall. Finally some detections on the brick wall are made by [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] and [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF]. The residuals of [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] on the second row are much closer to noise than the background, which amply justify the interest of detecting on the residual rather than on the background. Nevertheless, the residual has no reason to be uniform, as is apparent in the garage's residual. Even if the detections look any way acceptable, this nonuniformity of the residual noise suggests that center-surround detectors based on a local variance (as done in [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF]) might eventually be preferable. Fixing a target number of 10 -2 for the NFA means that under the (H 0 ) model, only 10 -2 false positives should occur per image. Yet, many of them shown examples show several false positives. Given the mathematical justification of these thresholds, false positives come from discrepancies between the hypothetical (H 0 ) model and the image. In the case of [START_REF] Zontak | Defect detection in patterned wafers using anisotropic kernels[END_REF], the over-detection in the trees of the picture with a man can be explained by the limited self-similarity of the trees: for this region, the nearest patches won't be close enough to the patch to reconstruct to fit the model, which requires at least one would-be-identical-except-for-the-noise patch in the neighborhood. The over-detection in the case of the undersea mine is likely a mismatch of the noise model with the picture noise. The many false alarms of this method for the other examples, makes us wonder if the model hypothesis is not too strong. The [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] method triggers many false detections in almost all examples tested. As we mentioned, this suggests that the Gaussian model for the detection pairs is inaccurate. This is not necessarily a problem for specific fault detection applications where the false alarm curves can be learned.

Quantitative evaluation

Estimating how well an anomaly detector works "in general" is a challenging evaluation task. Qualitative experiments such as the ones presented in section 9.4.1 give no final decision. Our goal now is to address the performance evaluation in terms of true positive rate (TP) and false positive rate (FP). To that aim, we generated a set of ten RPN textures [START_REF] Galerne | Micro-Texture Synthesis by Phase Randomization[END_REF] which are deprived of any statistical anomalies. We then introduced one artificial anomaly per rpn by merging a small piece of another image inside each of them. This was made by simple blending or by Poisson editing [START_REF] Pérez | Poisson image editing[END_REF] using the implementation of [START_REF] Di Martino | Poisson Image Editing[END_REF]. This method provides a set of images where a ground truth is known. Hence the detection quality measure can be clearly defined. Figure 9.5 shows one of the generated RPN images with an anomaly added and the anomaly's ground truth locus. Table 9.1 shows the result for our six methods on this dataset.

Table 9.1 demonstrates that for all methods, the predicted number of false positives (namely the theoretical NFA) is not always achieved. Indeed, the threshold for Table 9.1 was chosen so that the theoretical number of false detections per image should be 10 -2 . When taking into account the total number of pixels, this means that only around 4 × 10 -6 % false detections should be made by any method in this table. Only two methods are close to this number: [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF] and [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF], while the other compared methods make too many false detections. Such a false TP pixels (in %) FP pixels (in %) TP anomalies (in %) FP anomalies (in %) 33.1 1.79 × 10 -5 80 10

Table 9.1: Quantitative comparative results for anomaly detection. The number of true positive (TP) and false positive (FP) for different metrics is shown. TP pixels and FP pixels correspond to detections at a pixel level. A true positive is when an anomalous pixel is detected, and a false positive when a normal pixel is detected as anomalous. TP anomalies and FP anomalies evaluate if anomalies have been detected at all. A true positive is counted when there is at least one detected pixel in an anomalous region, and a false positive when there is at least one detection completely outside an anomalous region (with a maximum of 1 FP per image). These results were computed on a dataset of random uniform textures with a single anomaly added to each image. The thresholds were set for a targer number of false alarms (NFA) of 10 -2 per image (theoretical FP pixels of 4 × 10 -6 %). An example of an image from the dataset is shown in Figure 9.5. A method works correctly if it detects a high percentage of anomalies (third column) while having a good pixel coverage (first column), and a minimal false positive rate (second and fourth columns). Having a very low false positive rate is crucial for massive fault detection. In that sense, the best methods are [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF] and [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF].

positive target might seem too strict. Yet, it is an important requirement of anomaly detectors in fault detection to minimize the false alarm rate. Indeed excessive false alarms may put a production chain in jeopardy. Images are generally of the order of 10 7 pixels. Therefore if one wants to limit the false detection rate in a series of tested images, the false positive rate needs to be really small. The methods compared -except [MC14] -used the NFA framework. Therefore the discrepancy between the theoretical target and the obtained number of false alarms is explained by an inadequate (H 0 ) for the images. In fact, only the background model of [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF] matches completely these really specific textures that are RPNs.

To better compare the methods, we also computed ROC curves for all methods, Figures 9.7 and 9.8, as well as the table of true positive areas and false positive areas for a fixed positive rate of 1% (Table 9.2). The ROC curve aren't impacted by the choice of thresholds. Figure 9.7 is shown with a log scale for the number of false positives because its low or very low false positive section is much more relevant for anomaly detection than the rest. From these ROC curves and tables we can conclude, for this specific example, that [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF] (Area under the curve (AUC) 7.52) (which theoretically should be optimal for this problem) performs the best followed closely by [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] (AUC 7.03). It's worth noting that [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] is performing better than [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF] for very low false positive region. We then have [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] (AUC 5.79). The trailing methods . When available red is the detection with the threshold corresponding to the lowest NFA. For [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] we adopted a similar color coding: white between 0 and 0.5, cyan between 0.5 and 0.7, green between 0.7 and 0.9 and orange above 0.9.

Conclusions

We shall now address the objections that come to mind.

Is the method new? We have listed several saliency or anomaly detection methods based on rarity (sparsity, lack of similar patches, etc.) Thus involving sparsity or self-similarity is not new. The novelty of the method seems to be that it builds a new image, the residual, where the selfsimilar structure has been eliminated. As we have seen, using the a contrario framework is not new, but its use was restricted to detection in Gaussian colored noise.

Is the residual really stationary noise? The hypothesis is that the residual is noise, in the sense that it has lost all self-similarity. But this noise might not be stationary, which would lead to detection misses. Indeed, as a toy example assume that an image is composed of two textures, one very contrasted, and the other not. Then, the residual will have higher variance in the contrasted part. If the anomaly lies in the non-contrasted region, it might be missed because the (global) noise variance is overestimated (for this region). This can only be solved by localizing the detection, namely estimating the noise variance more locally, or equalizing its variance to make it stationary. This remains to be investigated.

Redundant detections?

We have shown the detector performance either on the color channels or on VGG feature channels. Observe that these detections can be fused by a mere union, as they are all meaningful. They are, as we saw, redundant and were shown to illustrate the independence of the method from the chosen channel.

Extensions. An extension to video is highly desirable but requires a computationally intensive implementation to perform nonlocal space-time patch comparison.

[BSGF10] has represented a breakthrough in the field. It brings a speed-up of almost two orders of magnitude over previous search techniques. The main reason for this is that PatchMatch performs all queries simultaneously and in collaboration, exploiting the fact that overlapping query patches are likely to have overlapping matches in the database image.

PatchMatch is an iterative algorithm which starts from a random initial guess for the k-ANNF and gradually refines it. In each iteration, each patch propagates its current candidates to its neighboring patches on the query image grid. Additionally, new candidates are searched randomly in the database image. This is performed by uniformly sampling in a series of concentric squares with decreasing radius centered at the position of the current best candidate.

Several works have improved the original PatchMatch algorithm reporting gains of one order of magnitude or even more. The main theme of these works is to combine PatchMatch with classical search structures to improve the random initialization or the random search. Coherency Sensitive Hashing (CSH) [START_REF] Korman | Coherency sensitive hashing[END_REF] uses locality sensitive hashing [START_REF] Indyk | Approximate nearest neighbors: towards removing the curse of dimensionality[END_REF] to improve both the propagation and the random search steps. KD-trees [START_REF] Louis | Multidimensional binary search trees used for associative searching[END_REF] are used in [START_REF] Olonetsky | TreeCANN -kd-tree Coherence Approximate Nearest Neighbor algorithm[END_REF] instead of the random initialization (after reducing the dimensionality of the patches). In [START_REF] He | Computing Nearest-Neighbor Fields via Propagation-Assisted KD-trees[END_REF] a KD-tree is used for the random search.

In this chapter we extend and generalize the theoretical framework of [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF] formalizing PatchMatch-like techniques. We apply it to study their convergence and give upper bounds on their convergence speed which are tighter than the ones of [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF]. In particular we study the original PatchMatch [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF], its generalization to k nearest neighbors [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF] and CSH [START_REF] Korman | Coherency sensitive hashing[END_REF]. Our estimates of a geometric convergence rate confirm the intuitions that led to the design of these algorithms. They also provide insight that might help improving current techniques.

We interpret PatchMatch in a general setting, as a collaborative optimization tool. While the results in [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF] consider only the original PatchMatch algorithm with one nearest neighbor, our results apply to k-nearest neighbors and to more complex PatchMatch algorithms, such as CSH [START_REF] Korman | Coherency sensitive hashing[END_REF], propagation-assisted KD-trees [START_REF] He | Computing Nearest-Neighbor Fields via Propagation-Assisted KD-trees[END_REF] and RIANN [START_REF] Ben-Zrihem | RIANN: Approximate Nearest Neighbor Fields in Video[END_REF].

We start by giving a precise definition of a generic PatchMatch algorithm in §10.2. In §10.3 we analyze the convergence of the generic PatchMatch, by bounding the probability of having energies higher than a given threshold after an iteration of the algorithm. In §10.4 we consider two specific algorithms, the original PatchMatch [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF] and CSH [START_REF] Korman | Coherency sensitive hashing[END_REF] and derive specialized bounds for them. Finally, in §10.5 we compare the theoretical bound to empirical cases. The proofs of our results can be found in the supplementary material.

Generic fast patch matching algorithm 10.2.1 Notations

We denote the query image by A and the database image by B. We write x ∈ A if x ∈ R d is a patch of the image A (with this abuse of notation, A is both an image and the set of its patches). The k-NNF from image A to B is denoted by ϕ. For a patch x ∈ A the matches associated to x in B are written as ϕ x , which is a set of k distinct patches of B. This means that the minimum value of U x over a set of k elements without repetitions is 0.

In practice, the L 2 distance in R d , written as . 2 , is often used to define as matching energy. To have a minimum value of 0 over k-sets, we write it as

U x (z) = x -z 2 -K x (10.2)
where if N k (x) ∈ B is the actual k th nearest neighbor of x in B, K x = x -N k (x) 2 . It follows that if u is a set of k distinct patches of B, then U x (u) 0. We write ϕ x to denote the worst match in this set, defined by

ϕ x = arg max y∈ϕx U x (y), (10.3)
where U x (.) is the energy function of Definition 10.1. The k-NNF ϕ assigns to each x ∈ A a k-set ϕ x that minimizes U x , i.e. such that U x (ϕ x ) = 0. The goal of PatchMatch algorithms is to find an approximate solution to this optimization problem. In the following we define a level-set, written {U z α}, for η a k-set of elements η ∈ {U z α} if and only if U z (η) α.

The following operator selects the k patches with smaller energy U x from a larger set.

Definition 10.2 (Merge operator). We define merge k x as an operator transforming a set of more than k patches into a set of exactly k patches satisfying:

|merge k x (u)| = k and ∀p ∈ u\merge k x (u), U x (p) U x (merge k x (u)). (10.4)
We use basic graph notation in the next sections. Consider a graph G = (V, E), where V is the set of vertices and E the set of edges. By y ∼ x we denote that y is parent of x, i.e. (y, x) ∈ E. We finally write µ(x) for the number of parents of a node x, i.e. A path of size m from x to z corresponds to a set of vertices (c i ) i=1,...,m ∈ V m such that c 1 = x, c m = z and for all i ∈ 1, m -1 , (c i , c i+1 ) ∈ E. We write P(x, z) for the set of all paths (of any size) from x to z.

Propagation graph

In a PatchMatch algorithm all patches in A collaboratively search for their matches in B. This collaboration is achieved through the propagation of candidate matches from each query patch to other patches in A following a specific order. This process can be described by defining a directed acyclic graph over the set of patches of A, which we call the propagation graph following [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF]. The vertices V in the propagation graph G are the set of patches of A, and the directed edges E describe the propagation relationships: (z, x) ∈ E means that z propagates matches to x. We associate an action function to each edge to allow for the possibility of applying a transformation to the propagated matches.

Definition 10.3 (Propagation action

). An action associated to an edge e ∈ E in the propagation graph, written A e , is a function which takes as argument a patch of an image and returns a patch. This patch can either be from the same image or be a new one generated by the action.

Definition 10.4 (Propagation graph). A propagation graph, written G, corresponds to a triplet (V, E, A) where V is the set of patches of image A, E a set of edges such that the graph is connected and acyclic and A are the action functions associated to each edge (Def. 10.3).

For matching image patches, the forward propagation follows the raster order from the top-left to the bottom-right. Each patch propagates matches to its neighbors on the right and down:

E = {(x, y) ∈ V ×V | y = right(x) or y = down(x)} .
(10.6)

The action shifts the candidate patch following the direction of propagation: if x ∈ A has a candidate z ∈ B, it propagates right(z) ∈ B to right(x). Thus, A(x, right(x)) = right(•), and analogously A(x, down(x)) = down(•). Some variants of PatchMatch add additional edges to this basic propagation graph seeking to enhance the impact of the propagation [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF][START_REF] Korman | Coherency sensitive hashing[END_REF]. These additional transitions connect pairs of patches in A that are similar, and thus good matches for one of patches in the pair are natural candidates for the other. The action associated with these new edges is simply the identity function.

PatchMatch algorithms have also been applied on meshes [START_REF] Klaudiny | Cooperative patch-based 3D surface tracking[END_REF][START_REF] Chen | Nonparametric texture transfer using meshmatch[END_REF]. In those cases, the propagation can be defined by considering a DAG on a subgraph of the mesh.

Random search

For the specification of a PatchMatch algorithm we need a mechanism for gathering random samples around the current candidates. Definition 10.5 (Random sampling operator). Given a database image B we define the transition kernel Q such that for any k-set ϕ of patches from image B, Q(ϕ, •) is a probability on the k-sets of patches from B. A set of k patches drawn from Q(ϕ, •) will be denoted by Sϕ, i.e. Sϕ ∼ Q(ϕ, •). We consider transition kernels with the property of having a non-zero probability of transitioning to a k-set of matches with arbitrary positive energy:

Q(ϕ, {U x < ε}) > 0,
∀ϕ, ∀x ∈ A, ∀ε > 0.

(10.7) Defined alongside the kernel Q is the worst case transition probability for an energy level ε at z ∈ A, as the highest probability of transitioning between two sets of patches with energy level higher than ε: C(z, ε) := sup

Algorithm

The generic PatchMatch is presented in Algorithm 28. It starts by initializing the propagation graph and the candidate matches. The propagation graph is initialized by defining a set of edges E and the associated propagation actions A.

The core of the algorithm is the iterative process that cycles through the nodes in the topological ordering updating the candidate matches according to the steps 5 and 8. The update in step 5 is the result of selecting the best k matches from a set of candidates given by: the current k matches ϕ n

x ; the matches A y,x ϕ n+1 y propagated from the parent nodes y ∼ x in the propagation graph; and samples S 2 A y,x ϕ n+1 y gathered around them. We call the latter set of samples the randomized propagation. In step 8 k random samples S 1 ϕ n+1/2 x around the new current matches are drawn to finish the update process.

The random samplings S 1 and S 2 are drawn from transition kernels Q 1 and Q 2 which might differ. The parent nodes y ∼ x precede x in the topological ordering, therefore they propagate to x the updated candidates ϕ n+1 y . After each iteration the propagation graph is reversed. The last step considers the possibility of modifying the propagation graph by adding/removing relevant/irrelevant edges. This is done for example in CSH [START_REF] Korman | Coherency sensitive hashing[END_REF]. In this section we study the convergence of the generic PatchMatch algorithm described in the previous section. We do so by upper-bounding the probability that the energy U x at a node x is larger than a threshold ε after an iteration of PatchMatch.

Point-wise energy decay

Our main result is Theorem 10.1 which bounds the decay of the probability of U x (ϕ n+1

x > ε). The propagation makes this probability smaller, since any of the ancestors of x could find a candidate that when propagated to x would yields an energy below ε. Indeed, as a consequence of the propagation, if ϕ x has energy higher than ε after a propagation pass, then the energies of the ancestors z of x need to be higher than a series of levels ε zx . The violation of any of these restrictions would cause the propagation of a candidate match to x with energy U x below ε. The higher these levels, the smaller the probability of not sampling random candidates with energies below them, and thus the smaller P (U x (ϕ n+1

x ) > ε). In Lemma 10.2 we show that imposing a lower bound ε of the matching energy of a node x results in lower bounds ε z,x for all its ancestors z that can be calculated recursively starting from x and following the reversed propagation ordering. In Theorem 10.1 we bound the probability that none of x's ancestors draw a random candidate with energy lower than the corresponding ε z,x . Lemma 10.2 (Constraints propagation). Consider an assignment ϕ n+1 resulting from an iteration of Algorithm 28. Then for each pair of nodes x, z ∈ V,

U x (ϕ n+1 x ) ε ⇒ U z (ϕ n+1 z ) ε z,x , (10.9)
where the levels ε z,x 0 are as follows. For the ancestors of x (i.e. P(z, x) = ∅) the levels ε z,x are defined via the following recursion starting from x and following the inverse propagation order:

     ε z,x = min U z (θ) θ ∈ y s.t. z∼y A -1 z,y ({U y ε y,x }) ε x,x = ε.
(10.10)

For the rest of the nodes ε z,x = -1.

Once we have established this "allowed" sets for the ancestors of x, the idea of the proof is to determine the probability of not escaping the allowed sets in any of the random searches.

Suppose z is an ancestor of x with a candidate ϕ n z ∈ {U z ε z,x } (for simplicity assume k = 1). The probability of keeping the energy higher than ε after one iteration decreases because there is a non-zero probability of taking a random sample outside this level set. The probability of sampling a candidate in an upper-level set is

P(Sϕ n z ∈ {U z l} | ϕ n z ∈ {U z l}) = 1 {Uz l} P(dϕ n z ) {Uz l} Q(ϕ n z , {U z l})P(dϕ n z ). (10.11)
If we knew the probability distribution of the candidate ϕ n x at iteration n, we could compute the above probability exactly. Instead, we bound it by assuming that all the mass of the distribution of ϕ n z concentrates on a single point: the one from which it is more unlikely to draw a sample with energy lower than l. The resulting probability is given by the worst case transition probability C in (10.8). This is the main intuition in the proof of our main result.

Theorem 10.1 (Point-wise convergence). Consider the field of candidate matches at iteration n, ϕ n . Define ϕ n+1 by applying an iteration of the Generic PatchMatch in Algorithm 28. Then, for all ε > 0, for all x ∈ A, we have

P(U x (ϕ n+1 x ) ε) P(U x (ϕ n x ) ε) z∈A C 2 (z, ε z,x ) µ(z) C 1 (z, ε z,x ) , ( 10.12)
where µ(z) was defined in (10.5) as the number of parents of node z and C i denotes the worst case transition probability for kernel Q i , as in Eq. (10.8).

For notation simplicity the product in (10.12) is over all patches z ∈ A, but the corresponding C i s are 1 for those z that are not ancestors of x in the propagation graph.

The result from Theorem 10.1 reflects some of the intuitive ideas that led the design of patch matching algorithms. Due to the propagation, all ancestors of x contribute to the probability of x of improving its energy. But not all nodes contribute the same to the bound. The larger ε z,x the better, as the C i (z, .) are non-increasing functions (Lemma 10.1). It is therefore important to design the propagation graph and its actions to maximize the ε z,x . The shift propagation actions introduced in [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF] in the patch matching application can be interpreted under the light of Theorem 10.1 as an heuristic to maximize the levels ε z,x .

Indeed, ε z,x is given by the minimum energy U z in the intersection of the sets A -1 z,y ({U y ε y,x }) for y in the set of nodes to which z propagates. Say y is the right neighbor of z. Then the propagation action from z to y is a right shift A z,y = right( • ). Suppose that the patches are of size s × s and the matching cost is a function of the sum of pixel-wise matching errors over the patch (such as any L p norm). Since the patches at z and y overlap, the energies U y (η) and U z (A -1 z,y η) have s(s -1) terms in common, and can expected to be similar. Therefore the minimum value of U z in A -1 z,y ({U y ε y,x }) should not be much lower than ε y,x . By assuming certain regularity conditions on the image it is possible to bound the difference between ε z,x and ε y,x . This is out of the scope of this chapter.

A result that is found in practice is that the rate at which the energy decreases becomes slower as the matches improve. Less contributions from other nodes are taken into account and mostly the random search improves the matching. This is in agreement with the theory, since as ε decrease, the sizes of the allowed set increase and the ε z,x decrease as well. According to Theorem 10.1, this results in a slower energy decrease.

We can also note that adding more edges to the propagation graph can only improve the convergence rate, requiring a smaller number of iterations to achieve a desired precision in the result (in probability). However, having more edges implies more computation each iteration. The optimal number of propagation links results from a trade-off between the computational effort per iteration and its impact on reducing the energy.

Most variants of PatchMatch use the same propagation graph and change the random search steps (including the initialization). This can have a dramatic effect of the convergence. In our bound the choice of the random search determines the coefficients C 1 and C 2 which depend directly on the search transition kernel. Two examples of random searches will be reviewed in §10.4.

In the case of the single nearest neighbor (k = 1) the bound from Theorem 10.1 can be improved. The improvement comes from a better version of the C i coefficients that considers the transition probability of merge k x (η ∪ S i η). The details are provided in the supplementary material.

Uniform decay and convergence in the mean

One of the advantages of PatchMatch algorithms is that the NNF converges rapidly as a whole. We now give bounds on uniform convergence and convergence in the mean.

Theorem 10.2. Consider the field of candidate matches at iteration n, ϕ n . Define ϕ n+1 by applying an iteration of the Generic PatchMatch in Algorithm 28. Then, for all ε > 0 we have

P( U . (ϕ n+1 . ) ∞ ε) P( U . (ϕ n . ) ∞ ε) z∈A C 2 (z, {U z ε}) µ(z) C 1 (z, {U z ε}) . (10.13)
Theorems 10.1 and 10.2, together with the assumption (10.7) on the transition kernels Q i , imply the convergence in probability of PatchMatch algorithms. Assumption (10.7) is necessary to ensure that C i (z, ε) < 1 for ε > 0. A stronger convergence can also be shown once we considered transition kernels Q i having this good property, the following result shows convergence in the mean, both point-wise and uniformly for the whole energy field.

Corollary 10.1. Assume that for any pair (η, ξ) of sets of k candidate matches Q

1 (η, ξ) > 0 (or Q 2 (η, ξ) > 0). Let (ϕ n ) be a sequence defined by Algorithm 28. Then ∀x ∈ A, E[U x (ϕ n x )] ---→ n→∞ 0 and E[ U . (ϕ n . ) ∞ ] ---→ n→∞ 0.
This assumption on Q i is reasonable when the universe of candidates is finite, such as when searching matching patches in a database image B. Some variants of PatchMatch have been used to minimize a field of functions over continuous parameters (scales and rotations [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF], planes in 3D [START_REF] Bleyer | Patchmatch stereostereo matching with slanted support windows[END_REF]). The previous corollary does not apply in these cases. Nevertheless both Theorems 10.1 and 10.2 remain valid even without assumption (10.7).

Specific PatchMatch algorithms

We now derive more specific bounds for two particular cases of the generic PatchMatch algorithm found in the literature.

The original PatchMatch algorithm

The original PatchMatch algorithm was introduced in [BSFG09a] for k = 1, and then generalized to k-nearest neighbors in [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF] (the heap algorithm) together with other generalizations. Most of them are covered by Theorem 10.1 as they are particular cases of Algorithm 28. In this section, we derive a more specific bound for one of these variants.

The field ϕ of k-matches is initialized at random, by uniform sampling from image B. The propagation graph is the basic one presented in §10.2.2, with the shift actions. For the forward propagation these are A(x, right(x)) = right(•), A(x, down(x)) = down(•).

We define S 1 by taking samples around the current best candidate (the RS best variant in [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF]). In [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF] these samples are drawn from a sequence of transition probabilities. The qth sample is sampled uniformly from a box of size [-dα q , dα q ] 2 centered at the current match, for q = 0, 1, . . . , q max , α = 0.5 and d = max{W, H} for a database image B of size W × H. The number of samples is chosen so that the smallest box is larger than a pixel. The first sample is taken uniformly on the whole image B, so that there is a positive probability of transitioning from and to any two patches in B. For simplicity, we assume that the random search operator S 1 ϕ n x draws k independent patches in B with the same probability distribution Q ( φx , .), where φx is the best current match. For all φ ∈ B, the support of Q (φ, •) covers all patches in B, so as in [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF][START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF], there is always a positive probability of transitioning between any two patches in B.

Without loss of generality, to simplify the notation we consider that the propagation graph is the same for all iterations (i.e. no alternation between iterations).

Proposition 10.1. The specific basic PatchMatch algorithm described above converges in probability to a NNF which minimizes the energy, namely

lim n→∞ P(U x (ϕ n x ) ε) = 0, ∀ε > 0, x ∈ A, (10.14)
with a geometric convergence rate. Moreover for all ε > 0, for all x ∈ A, we have that

P(U x (ϕ n+1 x ) ε) P(U x (ϕ n x ) ε) z∈A 1 -1 -C (z, ε z,x ) k , with C (z, α) := sup η Q (η, {U z a}).
For α > 0 we can guarantee that C (z, α) < 1.

Corollary 10.2. If k = 1 the upper bound can be written as

P(U x (ϕ n+1 x ) ε) z∈A C (z, ε z,x )P(U x (ϕ n x ) ε).
with C (z, α) := sup η∈{Uz a} Q (η, {U z a}).

Corollary 10.2 allows a direct comparison of our bound with the bound derived in [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF]. Both bounds have the same structure. However, the energy levels ε zx [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF] are smaller than ours causing a looser bound.

The CSH algorithm

Coherency Sensitive Hashing (CSH) was introduced in [START_REF] Korman | Coherency sensitive hashing[END_REF]. It uses Locality Sensitive Hashing (LSH) [START_REF] Indyk | Approximate nearest neighbors: towards removing the curse of dimensionality[END_REF] to improve the random search and to add propagation edges.

LSH is a method for nearest neighbors search based on partitioning the search space using a series of hash functions h drawn randomly from a family H. Each hash function partitions the space in "bins" containing points with the same hash value. The family H has the property that nearby points collide in the same bin with high probability, while far away points do so with smaller probability. This is made precise by the following definition 10.6. Definition 10.6. A family of functions H = {h : R d → U }, where U is the set of hashes, is called (R, cR, p 1 , p 2 )-sensitive if for any p, q ∈ R d (for simplicity of notation, we write P H ( .

) = P( . | H)) (1) if p -q R then P H (h(q) = h(p)) p 1 , (2) if p -q cR then P H (h(q) = h(p)) p 2 .
This is useful for nearest neighbors search when p 1 > p 2 .

Instead of using directly the elements from H, a second family of functions G, called an OR family, is created. The function g ∈ G is based on a set of n random functions h 1 , . . . , h n from H such that for all p, q, g(p) = g(q) if and only if there exist i ∈ 1, n such that h i (p) = h i (q). This will be the set of functions used to define the algorithm.

Lemma 10.3. If H is (R, cR, p 1 , p 2 )-sensitive then an OR family G created using n functions from H is (R, cR, 1 -(1 -p 1 ) n , 1 -(1 -p 2 ) n )-sensitive.
In CSH, hash functions drawn randomly at each iteration are used both to define the random search operator S 1 and the randomized propagation operator S 2 . Let us define the projection bin of a patch p ∈ R d as B g (p) = {q ∈ B | g(p) = g(q)}. (10.15) These projection bins depend on the random choice of the hash function, and are used to define the candidate set sampling operators. The random search S 1 is simply the projection bin of the query patch x, i.e. S 1 ϕ x := B g (x). The randomized propagation is defined as the union of the projection bins corresponding to the best b propagated candidates, where b k is a parameter of the method, that is:

S 2 A y,x ϕ y = ∪ b l=1 B g ([A y,x ϕ y ] l ), y ∼ x. ( 10

.16)

Here [A y,x ϕ y ] l , l = 1, . . . , k are the propagated candidates. Note that S 1 ϕ x does not depend on the candidate list, but only on the query patch x. This particularity, together with the properties of the hashing family H, can be exploited to derive a tighter upper bound than the one given in Theorem 10.1. CSH also uses hashing over the query image A to connect similar patches in the propagation graph. Our result is valid for any propagation graph. Proposition 10.2. For a (R, cR, p 1 , p 2 )-sensitive family of hashing functions such that R max z∈A K z (see Definition 10.1), the sequence (ϕ n ) defined by the CSH algorithm converges in probability to a minimizer of the total energy,

lim n→∞ P(U x (ϕ n x ) ε) = 0, ∀ε > 0, x ∈ A, (10.17)
with a geometric convergence rate. Moreover for all ε > 0, for all x ∈ A,

P(U x (ϕ n+1 x ) ε) P(U x (ϕ n x ) ε) (10.18) z∈A C 2 (z, ε z,x ) µ(z) f (p 1 , ε z,x ), (10.19
)

where f (α, β) = (1 -α k ) if β > 0, 1 otherwise.
In practice it is possible to compute an (R, cR, p 1 , p 2 )-family of hash functions such that p 1 is large enough to outperform the transition kernels used by the original PatchMatch algorithm. This explains the improved convergence rate of CSH reported in [START_REF] Korman | Coherency sensitive hashing[END_REF].

Experiments and discussion

We show experiments comparing the convergence predicted by the theory with the one found in practice, for the case of the original PatchMatch algorithm with k = 1 [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF]. The code to reproduce the experiments is available at https://github.com/pariasm/pm-bound.

Computation of theoretical bound

Given an energy value ε and a patch x in the query image A, the computation of the bound requires computing the energy levels ε z,x and the worst case transition probability C 1 (z, ε z,x ) for all ancestors z of x. The ancestors are all pixels located above and to the left of x during the forward propagation, and below and on the right of x during the backward pass. The original PatchMatch algorithm does not consider a randomized propagation S 2 A z,y ϕ n+1 y , thus in the following we will drop the subindex for C 1 and Q 1 .

The computation of the levels ε z,x is straightforward. We loop on the ancestors starting from x. Following the inverse propagation ordering we apply the recursion (10.10). The inverse actions A -1 z,y are one pixel shifts in the opposite direction to the propagation. For instance if z propagates to y at his right, then A -1 z,y {U y ε y,x } results from shifting to the left all elements of {U y ε y,x } ⊆ B.

Figure 10.1 shows examples of the resulting ε z,x at two image locations and for ε = 1 and ε = 15. The larger the ε z,x the smaller the coefficient C(z, ε z,x ) and the faster the convergence. We can see that the levels ε z,x at different points x can be very different. Moreover, increasing ε increases the levels ε z,x locally around x.

To compute the worst case transition probability C(z, ε z,x ), we compute for all η ∈ {U z ε z,x } the probability of drawing a sample from the upper-level set. As in the original PatchMatch, we take n independent samples following a sequence of uniform distributions S i η ∼ Q i (η, •) on square boxes centered at η with a decreasing sequence of radii. 1 For the worst case transition, we need the n samples to be in {U z ε z,x }, thus:

C(z, ε z,x ) = max η∈{Uz εz,x} n i=1 Q i (η, {U z ε z,x }). (10.20)
For each sampling radius r, the probability Q i (•, {U z ε z,x }) can be computed efficiently via a convolution of the indicator function of the upper-level set with a box kernel of size 2r + 1 × 2r + 12 . We use an integral image implementation to speed-up the computation.

Experimental validation

To validate the theory we contrast the predicted evolution of the matching energy with the empirical evolution. Since PatchMatch is a randomized algorithm, we estimate for a given energy level ε the probability that the energy is above ε for each iteration. The empirical probability is computed by running N iterations of PatchMatch a number of trials M . For a patch x ∈ A we define U n,m x the matching energy at iteration n of trial m. Then the empirical probability of U x ε is estimated as

P (x, ε, n) = P(U x (ϕ n x ) ε) = 1 M M m=1 1(U n,m x ε).
The theoretical bound on this probability is given by

B(x, ε, n) = P(U x (ϕ 0 x ) ε) • K(x, ε) n , ( 10.21)
where K(x, ε) = z∈A C(z, ε z,x ). The candidates in the original PatchMatch are initialized by sampling uniformly over the database image B, thus the initial probabilities can be easily computed as the area ratio between the upper-level set {U x ε} and the image domain B. In Figure 10.2 we plot the estimated probabilities P (x, ε, n) together with the theoretical bound B(x, ε, n) for a patch x ∈ A and several values of ε. As expected, the estimated probability is below the worst case bound. For smaller ε the bound becomes looser. For example: for the energy to be below than ε = 1 with a probability of 0.2, the theoretical bound predicts 178 iterations but in practice 60 were needed.

For k = 1, the gap between the theoretical bound and the empirical decay is mainly due to upper-bounding the transition probability (10.11) by the worst case probability C. To verify this we use a simplified random search: the sample Sη is taken uniformly over B (as in the initialization). In this way, we eliminate the dependence between the sample Sη and the current candidate η, and we have that P(Sϕ n z ∈ {U z l} | ϕ n z ∈ {U z l}) is the area ratio between the upper level set and the domain of image B, regardless of ϕ n z . We generate two images of random noise. The query image A is of size 24 × 24. It contains q = 20 × 20 = 400 patches of size 5 × 5. The database image B is of size 104 × 104, thus containing p = 100 × 100 = 10 4 patches. As before, we compute the empirical probabilities P and the bound for the bottom right patch. We show two results: one using only the uniform random search, and another one using the original centered random search. The plots correspond to an energy level of ε = 0.5, and only the unique global minimum is below ε. When using the uniform search, the empirical decay matches the theoretical prediction. The centered search shows a faster convergence. This experiment also shows that the theoretical bound captures the main intuition behind the design of the PatchMatch algorithm: if a region of image B is an exact copy of image A (or of a region in image A) it becomes very likely to find the copied region in the database image. Due to the propagation, as soon as one node in A finds its match, it will be propagated to all other nodes. In this case, one can compute exactly the probability that all the ancestors of x miss the copied region. The probability of missing the global optimum is 1 -1/p. The probability that all q ancestors of x miss the optimum is therefore (1 -1/p) q . This coincides with the theoretical bound.

As a final experiment we show results obtained computing C as an "average" transition probability instead of the worst case. This average transition results from assuming in (10.11) that ϕ n

x is distributed uniformly over the upper-level set. Figure 10.4 compares the predictions of this average C with the worst case C. While we do not have theoretical guaranties on the average C, its behavior that is much closer to the empirical case.

Conclusions

We presented a theoretical analysis of the convergence of PatchMatch algorithms. For an energy level ε, we show that the probability of having an energy above ε converges to 0 with the number of iterations, and we provide worst case bounds on the convergence rate. Our analysis applies to the case of k nearest neighbors, and to most variants of PatchMatch proposed in the literature. We give specific bounds for two of these algorithms: the original PatchMatch [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF][START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF] and CHS [START_REF] Korman | Coherency sensitive hashing[END_REF]. For the case of the original PatchMatch (with k = 1) we validate our results by comparing the predicted convergence rate with the one found in practice. The setting of our framework is rather general: the task of patch matching is viewed as an optimization problem where the goal is to minimize several non-convex energies U x over the same domain. This might allow the application of these techniques to similar optimizations problems in other areas. For the case of matching patches, it would interesting to precise the link between the regularity of the images and the convergence rate, at least for certain simple models of images.

Detection of copy-paste forgeries based on PatchMatch

This chapter presents an implementation and discussion of the recently proposed 'Efficient Dense-Field Copy-Move Forgery Detection' by Cozzolino et al.. This method is a forgery detection based on a dense field of descriptors chosen to be invariant by rotation. Zernike moments were suggested in the original article. An efficient matching of the descriptors is then performed using PatchMatch, which is extremely efficient to find duplicate regions. Regions matched by PatchMatch are processed to find the final detections. This allows a precise and accurate detection of copy-move forgeries inside a single suspicious image. We also extend successfully the method to the use of dense SIFT descriptors and show that they are better at detecting forgeries using Poisson editing.

Introduction

Copy-move forgeries correspond to the case where a region is copied from the image and then pasted in another position of the same image. The copy could be modified in the meantime, but the detection presented here is only guaranteed to work when a rotation has been applied. Nevertheless we shall see that this method is also mildly robust to other perturbations such as noise. This type of forgery, which can easily be performed with most image editing software tools, is usually used to hide undesired details in the image, or to add new details. Detecting such modifications can be challenging, especially when carried out by a professional image editor. Many different methods have been developed to detect such modifications. In [START_REF] Farid | Image forgery detection[END_REF], Farid presented a large number of such methods: from the simplest ones such as pixel-based methods to much more complex physics-based methods by which, for example, the illumination of the scene is studied. In this study, we focus on a single family of detection methods. These methods, specialized im detecting copy-move forgeries, are organized in the following way. First, descriptors are computed on the suspicious image. These descriptors are chosen to represent well the local behavior of the suspicious image. They are then matched to each other during a matching step to detect abnormal similarities within the image. In the final step, post-processing is applied to refine detections and diminish the number of false positives. The first papers on this subject proposed classic sparse image descriptors such as SIFT [START_REF] Pan | Region duplication detection using image feature matching[END_REF] or SURF [START_REF] Bl Shivakumar | Detection of region duplication forgery in digital images using surf[END_REF]. A recent benchmark [CRJ + 12] of many different methods based on both sparse descriptors and dense descriptors shows that dense descriptors largely outperform sparse descriptors. For this reason [START_REF] Cozzolino | Efficient dense-field copy-move forgery detection[END_REF] chose to use dense descriptors. Matching the descriptors, or equivalently finding the nearest neighbor for each descriptor, is a well studied problem but actually quite hard. Many different approaches have been developed but all actually compute approximate matches. The approximation can be computed using partition trees such as the well known KD-trees [START_REF] Louis | Multidimensional binary search trees used for associative searching[END_REF], or VPtrees [START_REF] Peter N Yianilos | Data structures and algorithms for nearest neighbor search in general metric spaces[END_REF], or adapted hashing functions such as LSH [IM98,GIM + 99]. A recent breakthrough for image specific algorithms is PatchMatch, developed by Barnes et al. [BSFG09a,[START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF]. This algorithm relies on the structural properties of the image and multiple sampling, which in practice performs extremely well on dense descriptors. It is also the matching algorithm adopted by the reviewed method.

The rest of the chapter is organized as follows: the descriptors used for the processing, Zernike moments, are presented in Section 11.2. The matching algorithm PatchMatch and the modifications implemented to improve the matching for our problem are presented in Section 11.3. Several pre-and post-processing steps presented in Section 11.4 are then applied to obtain precise and accurate detections. Finally, a couple of experiments where we show success and limits of the method are presented in Section 11.5, where we successfully extend the method to dense SIFT descriptors..

Zernike Moments

The absolute values of Zernike moments are rotation invariant descriptors that are adapted to the task of detecting image repetitions up to a rotation. We address in this section their efficient computation by the Xin et al. [START_REF] Xin | Accurate computation of zernike moments in polar coordinates[END_REF] method. For a patch u of an image I (defined for now as continuous on [-1, 1] 2 ), the Zernike A nm moment of order n and angular dependence m, for m < n and m ≡ n[2], is defined by

A nm = n + 1 π D u(x, y)V nm (x, y)dxdy, ( 11.1)
where V nm is the complex Zernike polynomial of order n and angular dependence and A nm = 0 otherwise (therefore in the following study, the case m ≡ n[2] will be omitted). The polynomials are defined over the unit disk D(0, 1) and usually expressed in polar coordinates ρ and θ. They are fully defined by V nm (ρ, θ) = R nm (ρ)e imθ , (11.2)

R nm (ρ) = n-m 2 s=0 (-1) s (n -s)!ρ n-2s s!( n+m 2 -s)!( n-m 2 -s)! . ( 11.3)
Zernike moments can be interpreted as scalar products between the image u and the Zernike polynomials. The absolute value of these moments is rotationally invariant. Indeed, consider a rotation of angle α of the given image, or equivalently, a rotation of angle α of the Zernike polynomials V α nm = V nm e iα , the studied moment A α nm would then be and therefore

A α nm = n + 1 π D u(x, y)V α nm (x, y)dxdy (11.4) = n + 1 π D u(x, y)V nm (x,
|A α nm | = |A nm |, (11.8)
which confirms the rotation invariance of the magnitude of these moments.

In our case, the moments will be computed on each patch of the image (considered as a small image itself). Within the same framework presented previously, one can also define translation invariant descriptors by computing a dense field of descriptors over the entire image. The computation is trickier than it appears at first sight because the image (noted u and so far considered continuous) is sampled over a Cartesian grid whereas Zernike polynomials are more naturally expressed in polar coordinates. The sampled version of I and u with a sampling step 1 sp will be noted respectively Ĩ and ũ. The coordinates (x 0 , y 0 ) denote the center of ũ in Ĩ and ũ is defined on -sp, sp . The easiest solution would be to estimate the integral from Equation (11.1) by sampling over the Cartesian grid by

A nm = sp sx=-sp sp sy=-sp ũ(s x , s y )V nm ( s 2 x + s 2 y , atan2(s y , s x )). (11.9)
Xin et al. have shown in [START_REF] Xin | Accurate computation of zernike moments in polar coordinates[END_REF] that this approximation suffers from multiple drawbacks and is not properly rotation invariant. The rotational invariance being one of the most important features of the Zernike moments in our case, we shall use the more accurate estimation proposed in [START_REF] Xin | Accurate computation of zernike moments in polar coordinates[END_REF]. First, we need to resample the image. Indeed the original Cartesian sampling is not adapted to the unit disk. Figure 11.1a shows that whatever the choice of pixels taken to do the integration on, it will never truly form a disk; therefore leading to approximation in the computations. An adequate image partitioning is required to care for the rotation invariance. First, the unit disk is split into K regions along the radial direction. These regions are then divided into "pixels" with the same area. For this reason each region is split into (2k + 1)L subregions. This sampling of the unit disk is presented in Figure 11.1b. It has the nice property of not losing resolution compared to the original image. In addition, the new pixels have approximately the same area as the old ones. The image is considered piecewise constant over these "pixels". The value over such a pixel, noted f (ρ kl , θ kl ), is computed by image interpolation. Xin et al. recommended the usage of a bicubic interpolation. This yields We can now express Zernike moments in polar coordinates by a simple change of variable,

f (ρ k , θ kl ) = ρ k cos(θ kl ) +2 sx= ρ k cos(θ kl ) -1 ρ k sin(θ kl ) +2 sy= ρ k sin(θ kl ) -1 ũ(s x , s y )h (ρ k cos(θ kl ) -s x ) h (ρ k sin(θ kl ) -s y ) , (11.10) where h(x) =        3 2 |x| 3 -5 2 |x| 2 + 1 if |x| ≤ 1 -1 2 |x| 3 + 5 2 |x| 2 -4|x| + 2 if 1 < |x| ≤ 2 0 otherwise . ( 11
A nm = n + 1 π 2π 0 1 0 u(ρ, θ)V nm (ρ, θ)ρdρdθ. ( 11.12)
This leads to the following estimate of A nm by sampling the integral over the newly created regions

A nm = n + 1 π k l f (ρ k , θ kl )w nm (ρ k , θ kl ), (11.13)
where w nm corresponds to

w nm (ρ k , θ kl ) = Ω kl R nm (ρ)e -imθ ρdρdθ (11.14) = ρ k,2 ρ k,1 R nm (ρ)ρdρ θ kl,2 θ kl,1
e -imθ dθ , (11.15) where the support of the pixel Ω kl is parameterized by ρ k,1 , ρ k,2 , θ kl,1 , θ kl,2 (see Figure 11.2).

Besides the two previous integrals can be computed in closed form

ρ k,2 ρ k,1 R nm (ρ)ρdρ =    n-m 2 s=0 (-1) s (n -s)! ρ n-2s+2 k,2 -ρ n-2s+2 k,1 (n -2s + 2)s!( n+m 2 -s)!( n-m 2 -s)!    , ( 11.16)
and Xin et al. have shown that computing the moments using this framework allows for a much better rotation invariance compared to the regular Cartesian approximation. The sums in Equation (11.13) can be reordered to compute efficiently the entire set of Zernike moments for the 208 entire image Ĩ. Indeed A nm can be rewritten

θ kl,2 θ kl,1 e -imθ dθ =    θ kl,2 -θ kl,1 if m = 0 i m e -imθ
A nm = n + 1 π k l f (ρ k , θ kl )w nm (ρ k , θ kl ) = n + 1 π k l sx sy ũ(s x , s y )h (ρ k cos(θ kl ) -s x ) h (ρ k sin(θ kl ) -s y ) w nm (ρ k , θ kl ) = n + 1 π sx sy k l ũ(s x , s y )h (ρ k cos(θ kl ) -s x ) h (ρ k sin(θ kl ) -s y ) w nm (ρ k , θ kl ) = n + 1 π sx sy ũ(s x , s y ) k l h (ρ k cos(θ kl ) -s x ) h (ρ k sin(θ kl ) -s y ) w nm (ρ k , θ kl ) = n + 1 π sx sy ũ(s x , s y )F nm sx,sy , (11.18)
which can now be used to define an "image" of zernike moments such that

A nm (x 0 , y 0 ) = n + 1 π sx sy ũ(s x , s y )F nm sx,sy = n + 1 π sx sy Ĩ (x 0 + s x , y 0 + s y ) F nm sx,sy . (11.19)
Equation (11.19) shows that the A nm s can be computed efficiently by convolving the image Ĩ with the newly defined filter F nm . A null extension is chosen for the convolution for efficiency and simplicity. Indeed, the moments computed on the border won't have any of the properties required for the matching, whatever the extension (in particular invariance to rotation is the hardest to ensure); therefore they should be discarded anyway in the following steps and the choice should be made only for computation efficiency. The pseudocode for the full computation of the dense field of moments for a given image is given in Algorithm 19. It yields both translation and rotation invariant descriptors with which copy-move forgeries will be detected. The two versions of Zernike moments will be compared for the problem of forgery detection with rotation in Section 11.5.1.

Patchmatch

In order to detect forgeries, we'll have to find the nearest neighbors of the descriptors from Section 11.2. Usually finding nearest neighbors in high dimension is a rather difficult problem but, for the specific case of images, the PatchMatch heuristic is faster and more efficient than nonspecific heuristics. Instead of using the original PatchMatch presented in Chapter 10, we use an improved version that has a faster convergence. In a follow-up paper [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF], the authors of PatchMatch proposed several other versions of the algorithm to either solve slightly different but similar problems or to improve the speed and/or the quality of the matching. In particular, they considered the problem of matching modulo rotations and scaling. While the modification for rotation and scaling yields a slow convergence rate in practice (the space of possible matches becoming far too large for the random search), and is not adapted to rotation invariant descriptors (it explores the entire space of rotations), we will use the modifications suggested when applying PatchMatch to the same image. The idea behind this modification is to try to take advantage of the natural symmetric consistency within the matches. Indeed, if b is the current candidate as nearest neighbor of a, then a is a good candidate for the nearest neighbor of b.

Algorithm 29: Zernike moments computation input : An image I, a half-size of patch sp, a maximum order o output : This will be combined with the modifications suggested by Cozzolino et al. in [START_REF] Cozzolino | Efficient dense-field copy-move forgery detection[END_REF] to take rotation invariant descriptors into consideration and improve their matching. In order to increase the quality of the matches -when taking rotations into consideration -two more propagations are added. These two propagations correspond to a propagation from the top-left patch as well as from the top-right one. This means that information is propagated to all connecting patches. These propagations are represented in green in Figure 11.3b. A second set of propagations, called firstorder propagations, are also added to the list of propagations. These propagations are different from the previous ones because they are not simply a displacement propagation. They propagate linearly varying offset modifications. In order to simplify the notations for some of the equations, we also introduce the optimal displacement s alongside an ANNF f such that f(a) = a + s(a). In the following the ANNF and the optimal displacement will be used interchangeably, the one leading to the simplest equations will be chosen. For two functions f and g, the composition of functions is written f • g = f (g(.)). Using a rough Taylor expansion, an approximation of a function f can be made

A dense descriptor map Z 1 foreach (n, m) s.t. n ∈ 1, o , m ∈ 1, n and n ≡ m[2] do 2 for s = 0 to n-m 2 do 3 C nms ← (-1) s (n-s)! (n-2s+2)s!( n+m 2 -s)!( n-m 2 -s)! //
f (x + 1) ≈ f (x) + df (x) dx ≈ f (x) + f (x) -f (x -1) 1 ≈ 2f (x) -f (x -1). ( 11

Forgery Detection

Once the displacement map, a function that for each patch of the reference image, associates its nearest neighbor, has been computed using PatchMatch, forgeries can be detected. In order to avoid false alarms, multiple pre-processing and post-processing techniques are applied. The different steps for the detection are:

1. Application of a median filter.

2. Computation of an error map and early detections.

3. Remove detections that are too small. 4. Remove detections that are too close. 5. Symmetrize the detections.

Dilate the detections.

The complete algorithm performing forgery detection is presented in Algorithm 37. All these steps will be developed and explained in the following sections. It is also possible to detect internal copies where the copies have been flipped. It only requires to compute descriptors for the flipped patches; this is easily done by computing descriptors on the flipped image and flip the resulting descriptor map. The distance function used for PatchMath in this case is the distance from the regular descriptors to the flipped version. Another possible solution is to use flip-invariant descriptors such the one presented in [START_REF] Zhao | Flip-invariant sift for copy and object detection[END_REF] or [START_REF] Xie | Image classification with max-sift descriptors[END_REF] so to avoid two computations of the descriptors; this is not studied in this chapter though.

Median Filter

The first step is to apply a median filter to the displacement map. It smooths the displacement and improves the detection rate by reducing the overall error computed in the following step. The algorithm used for the median filtering is shown in Algorithm 31.

Error Filter and Detection

A copy-move forgery (as its name indicates) copies a region of the image, possibly rotates it, and pastes it at a different image position. Both regions (the original one and the forged one) can then be matched by an affine transform. The idea presented in [START_REF] Cozzolino | Efficient dense-field copy-move forgery detection[END_REF] is then to compare the field of matches created using PatchMatch to the one computed using the best affine transform based on the field of PatchMatch matches.

Let P ∈ M 3×n (R) be the homogeneous coordinates of a set of points. We set = ∆ -∆P T (P P T ) -1 P 2 = ∆(I -H) 2 , where H = P T (P P T ) -1 P. (11.26) The set of points P used to estimate this error will be a disk of radius ρ e and the error will be associated to the center of the disk, which delivers an actual error map. In practice the result is independent of the coordinates in P , as long as ∆ is set conveniently. Therefore H can be precomputed for all regions. The error detection map is given by thresholding by τ the error map estimated with Equation (11.26). The features and the error filter aren't properly defined on the boundary (it would require to define the behavior of these objects outside the image as well), therefore detections on the boundary are not well defined either. This is why the boundary of the image is removed during the detection step so as to avoid any false detection due to a false matching. The algorithm used to compute the error detection map is summarized in Algorithm 32.

Algorithm 32: errorDetectionFilter algorithm input : A displacement map D, the radius of the filter ρ e , τ a threshold, sp the half-size of the patch used to compute the features output : An error detection map E 1 P ← D(0, ρ e ) // Define the homogeneous coordinates 2 H ← P T (P P T ) -1 P // See Equation (11.26 The parameters that will be used for the experiments are those suggested in [START_REF] Cozzolino | Efficient dense-field copy-move forgery detection[END_REF]. They are reminded in Table 11.1. The different images used in this section come from the FAU database presented by Christlein et al. in [CRJ + 12]. The first example, shown in Figure 11.5, is a straightforward example illustrating the quality of the detections by the method in the simplest case. We can see that every step performs as expected and provides an accurate estimation of the forged regions. It is also important to test the method's resilience to false positives. This is why we applied the exact same method to an intact image containing several instances of a similar object, shown in Figure 11.5. The result confirms the absence of detection in this case even though some regions could have appeared forged due to their similarity. One of the requirements of the method was to be rotation resilient. An example of forgery with a rotation is presented in Figure 11.5. It actually confirms that the detection succeeds when a rotation has been applied. Even though only a single example is shown, more tests confirming the rotation resilience have been done with varying degrees of rotation. The same experiments have also been done using basic Cartesian Zernike moments in order to see the importance of the resampled version presented in Section 11.2. In this case, also presented in Figure 11.5, the detection is much worse (only a much smaller region is detected) than when the resampled Zernike moments are used.

We also tested other classic perturbations such as the addition of noise. Figure 11.6 presents a result with a small amount of noise added after the forgery. The method still works in presence of small noise. The addition of a large noise in a forgery would make the image suspicious anyway. Figure 11.6 shows the importance of also taking account flipped copies. The next example of Figure 11.6 is no longer a simple copy-move forgery but an actual editing of the image. Indeed the copied section was integrated using the Poisson editing method presented in [START_REF] Pérez | Poisson image editing[END_REF] and implemented in [START_REF] Di Martino | Poisson Image Editing[END_REF]. Poisson editing is quite adapted to forgery as it blends nicely the pasted region in the image. This time, the algorithm actually fails. This shows the limitation not of the method itself but of the descriptors, namely the amplitude of Zernike moments. Those descriptors achieve the wanted invariance but at the same time they are extremely rigid. With appropriate descriptors, the exact same methodology could be applied and should yield good detections. Removing the Zernike moment of order 0 from the list of descriptors used (which actually corresponds to the mean of the local region) didn't improve the results for Poisson editing.

Another limit of the method is the presence of multiple copies. In such a case, the PatchMatch displacement field is at risk of splitting between the multiple copies, which increases the error and decreases the chance of an actual detection. Figure 11.6 shows an example where this problem arises. In that case, parts of each of the copies are actually detected but none is entirely detected for the reasons explained above.

Using Dense SIFT Descriptors

Although the method performs as expected, as shown in Section 11.5.1, it is disappointing that forgeries such as Poisson editing are not detected. For this reason, the same experiments have been repeated after changing the Zernike moments by dense SIFT descriptors. SIFT descriptors, presented by Lowe in [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF], have been shown to be state of the art descriptors in image matching thanks to their invariance to noise, changes of lighting or to small changes of viewpoint. In particular, they are rotation invariant, just like the Zernike moments. A dense version of these descriptors can be computed by considering all patches instead of those marked by points of interest. The dense version was the one used for the experiments. The VLFeat implementation of dense SIFT from [START_REF] Vedaldi | VLFeat: An open and portable library of computer vision algorithms[END_REF] was used. The parameters are listed in Table 11.2. Figures 11.7 .8: Result of the method using dense SIFT on (from top to bottom): a forged image with a forgery applying a translation on top of which a noise of standard deviation 25 was added, a forged image for a forgery applying a Poisson editing, a forged image for a forgery applying translations when a same object is copied at multiple different positions.

Conclusion

The forgery detection method analyzed in the present chapter proves to be an efficient detector. One of its major advantages is its resilience to rotations. It is also robust to image perturbations such as the addition of noise. Nevertheless, it is not deprived of drawbacks. In particular, the results are particularly disappointing when considering multiple copies of an object or when the forgery has been performed by a classic copy-paste method such as Poisson editing. A better choice for the descriptor should be able to fix this deficiency of the method though. Dense SIFT descriptors were considered as an alternative to improve the detection of forgeries performed by Poisson editing. Even though these descriptors allow for better detections, they are more likely to produce false positives. Nevertheless, this shows that the method can be modified and improved by varying the descriptors. The only drawbacks found -which are mainly linked to the usage of PatchMatch -is the inability of the method to detect multiple copies but also to not distinguish well similar objects from actual copies.

12 Detection of copy-paste forgeries based on sparse descriptors

Detecting reliably copy-move forgeries is difficult because images do contain similar objects. The question is : how to discard natural image self-similarities while still detecting copy-moved parts as being "unnaturally similar"? Copy-move may have been performed after a rotation, a change of scale and followed by JPEG compression or the addition of noise.

For this reason, we base our method on SIFT, which provides sparse keypoints with scale, rotation and illumination invariant descriptors. To discriminate natural descriptor matches from artificial ones, we introduce an a contrario method which gives theoretical guarantees on the number of false alarms. We validate our method on several databases. Being fully unsupervised it can be integrated into any generic automated image tampering detection pipeline.

Introduction

Photo and video editing includes the insertion or removal of parts of the image, often performed by internal or external copy-move operations. The Poisson editing technique [START_REF] Pérez | Poisson image editing[END_REF][START_REF] Di Martino | Poisson Image Editing[END_REF] allows for seamless insertions and is now routinely used for special effects in movies, in software like Photoshop or in popular mobile phone applications. Most editing operations are driven by aesthetic goals. Yet their usage can easily become malicious and help forging false evidence, fake news, or alter results in scientific publications [START_REF] Bik | The prevalence of inappropriate image duplication in biomedical research publications[END_REF].

It is therefore of primary importance to provide public and professionals with reliable scientific tools detecting traces of any intentional alteration of a photograph. Several different techniques are relevant here: image splicing (internal or external) can be detected through its local alterations of the compression encoding and of the JPEG blocks [LHTT09,CGFY12,NGvGCM18], its inconsistent demosaicking traces [PF05, FBDRP12, BGvGM18], or directly [START_REF] Ng | Blind detection of photomontage using higher order statistics[END_REF][START_REF] Hsu | Image splicing detection using camera response function consistency and automatic segmentation[END_REF][START_REF] Huh | Fighting fake news: Image splice detection via learned self-consistency[END_REF]. Methods tracking other features such as noise inconsistencies, lightning inconsistencies, chromatic aberration inconsistencies, etc. were listed in the broad review [START_REF] Farid | Image forgery detection[END_REF].

This chapter focuses on a specific type of image splicing called "copy-move". As its name indicates it consists in copying a region of the image and pasting it somewhere else. Rotation, scaling, change of contrasts and other manipulations are sometimes applied to the piece being copied before pasting it. The method can be used to replicate objects, but sometimes also to hide an object by a texture borrowed elsewhere in the image. Copy-move detection methods can be divided into two main categories: Block-based and keypoint-based. The block-based approaches try to match regions by blocks. In order to match the blocks more easily and more efficiently it is frequent to represent the block in a compact form by dimensionality reduction, e.g. with PCA, DCT [START_REF] Cao | A robust detection algorithm for copy-move forgery in digital images[END_REF] or DWT [START_REF] Li | A sorted neighborhood approach for detecting duplicated regions in image forgeries based on dwt and svd[END_REF]. The compact representation may also ensure the invariance of the detection to rotations by using Zernike moments [START_REF] Cozzolino | Efficient dense-field copy-move forgery detection[END_REF][START_REF] Ehret | Automatic Detection of Internal Copy-Move Forgeries in Images[END_REF] or a similarity invariance with the Fourier-Mellin transform [START_REF] Bayram | An efficient and robust method for detecting copy-move forgery[END_REF][START_REF] Li | Rotation robust detection of copy-move forgery[END_REF]. These methods generally manage to detect the forged regions, but are computationally demanding. Instead of directly trying to match blocks, featured-based methods compute sets of keypoints and then match these keypoints. Many of these methods are based on SIFT [ABC + 10, ABM10] or SURF [START_REF] Bo | Image copy-move forgery detection based on surf[END_REF]. The descriptors associated to the keypoints are invariant to rotation, scaling and even moderate affine distortions. Yet, precisely out of too much robustness, these methods may cause false detections when similar objects are present in an image. As argued in [WZS + 16], most methods therefore suffer from a false positive problem caused by the occurrence of "natural" self-similarity. This is the problem that we attempt to tackle here.

Section 12.2 introduces our method and Section 12.3 shows experimental results on different datasets. Finally perspectives are going to be presented in Section 12.4.

Copy-move matching with SIFT-like matching

Like in the SIFT algorithm [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF] we start by computing a set of sparse keypoints. These keypoints are usually located in textured regions. Then a descriptor is associated to each of these keypoints. Finally the descriptors are matched to each other to define the detection. These three steps are summarized in the next paragraphs.

Keypoints

The keypoints correspond to the extrema of the normalized Laplacian scale-space. In practice they are computed using differences of Gaussians. The positions of the maxima are then found for each scale. To each keypoint is associated a scale and a principal orientation. A more detailed analysis can be found in [START_REF] Otero | Anatomy of the SIFT Method[END_REF].

Descriptors

This first, classic, SIFT step gives a list K = (k i ) of keypoints. From each of these keypoints (consisting of a spatial position, a scale and an orientation), a square patch p i of size (N + 2) × (N + 2) can be sampled. The gradients in both directions are then computed from these patches yielding an N × N gradient patch D i with vector values ( ∂p i ∂x , ∂p i ∂y ). Contrary to SIFT, we keep these matrices for the matching step. Indeed, using histograms of gradients (HOGs) to represent the gradient patch would be too robust a representation and lead to the detection of natural repetitions. Hence, following [START_REF] Grompone | A contrario patch matching, with an application to keypoint matches validation[END_REF] and [START_REF] Rodríguez | Affine invariant image comparison under repetitive structures[END_REF]. we encode the key point k i by its gradient patch D i . This allows for an invariance to uniform illumination changes. In SIFT, the descriptors are computed on a grayscale version of the image for matching applications. In the forgery case, it is interesting to consider all information available. So our gradient descriptors keep three channels, one for each color. For simplicity our matching step will be presented using a grayscale descriptor, but extends immediately to color as well. Color descriptors will be used for the experiments in Section 12.3.

Matching

Two naturally similar objects are rarely exactly similar. This is because there are always differences in their illumination in a real scene, and physical differences that do not necessarily catch the eye, in addition to the acquisition noise. Our matching process takes advantage of these serious variations to discriminate between similar objects and digital copies. Consider two keypoints k i and k j located respectively on the original object and on the forged copy, so that they match with a regular matching method such as SIFT [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF] or [START_REF] Rodríguez | Affine invariant image comparison under repetitive structures[END_REF]. In that case the descriptors D i and D j associated to these keypoints should be exactly the same, namely ∀k, l ∈ {1, . . . , N }, D i k,l = D j k,l . Of course this perfect quality is not reached in practice. Several copy-move steps could introduce small differences such as: the interpolation due to a rotation or zoom or even a post-processing step such as the addition of noise and/or a compression after forgery. Nevertheless, we can enforce a very close match between each part of the descriptors by an exigence like ∀k, l ∈ {1, . . . , N }, D The key question is to fix the right detection threshold τ , to have a matching criterion that rejects genuinely similar objects while still detecting well copy-move forgeries. This threshold can be computed rigorously using the a-contrario theory [START_REF] Desolneux | From gestalt theory to image analysis: a probabilistic approach[END_REF] which is a probabilistic formalization of the non-accidentalness principle [START_REF] Lowe | Perceptual organization and visual recognition[END_REF]. This principle has shown its practical use for detection purposes such as segment detection [START_REF] Grompone | Lsd: A fast line segment detector with a false detection control[END_REF], vanishing points detection [LGv-GRM14] and anomaly detection [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF]. The a-contrario theory provides a way to compute automatically detection thresholds while having a control on the number of false alarms (NFA). It replaces the usual p-value by drawing into account the number of tests and therefore controlling the overall number of false alarms in a given detection task. The method only requires a simple texture the forged region has the easier it is to detect. This is because a textured region will generate more keypoints and therefore will increase its chances of matching.

Figure 12.3 shows several failure examples. Most failures come from the fact that the method can't deal with more severe distortions such as a tilt or non-uniform illumination change. The method also fails to detect flat regions, as no keypoints are computed on these regions. As for the false detection, it does not contradict the a contrario model. We requested at most = 1 false detection per 100 images with the threshold given in Section 12.2.3, and we found one with 200 images tested for the COVERAGE dataset.

Perspectives

In this chapter we have presented an unsupervised method to detect copy-move forgeries that is not only invariant to rotation, scaling and global change of illumination, but also robust to the presence of similar but genuinely different objects or regions. The method, being parameterless and very fast, can be included in the necessary long series of tampering tests applied to a suspicious image.

The limits of the method are closely linked to its strength. Because it is robust to the presence of naturally similar objects, it is less reliable in case of large degradation of a copied digital ones. We nevertheless found that the method is robust enough to usual noise and compression levels. An image that has been degraded too much is suspicious anyway, since nowadays the quality of an image taken with a mobile is very good. Thus only images with a good enough quality should be tested. Highly degraded images would be anyway suspicious regardless of any such more sophisticated examination. The second limit is the usage of sparse keypoints. These keypoints are only computed in regions that are contrasted enough (non-flat areas) which means that forgeries in these regions might not detected. Finally matching keypoints give anchor points and do not delimit forged regions precisely. A natural extension of the method would be to extract the forged regions from the anchor points while still keeping a good control over the number of false detections. Finally coupling the method with a noise estimator could arguably make it still more discriminant.

Conclusion

The main focus of this dissertation was video denoising, with detours to other problems that can be approached by applying some of the same techniques used for video denoising. We first studied patch-based methods, which were the state-of-the-art at the beginning of the thesis. We looked at many different aspects of patch-based methods: the models for similar patches, patch shapes and the patch search. We took an in-depth look at a popular video denoising method called VBM3D in Chapter 1. We also looked at how to integrate new improvements such as multi-scale, optical flow and spatio-temporal patches. In particular, VBM3D combined with spatio-temporal patches and a patch-search guided with an optical flow gives very competitive results even when compared to the latest state of the art, especially for higher noises. VBM3D and its variants provide an interesting trade-off between denoising quality and computational complexity.

We then turned our attention to the patch search, a rather overlooked aspect of patch-based methods for video. We studied the performance gain obtained by expanding the local patch search into a global one for patch-based video denoising algorithms in Chapter 2. With the global search, the patches found can follow long trajectories in the video, thus fully benefiting from the temporal redundancy of videos. Thanks to an efficient approximate search, we proposed global video denoising methods by extending to video BM3D and NL-Bayes, two patch-based image denoising algorithms. We obtained a significant boost on the denoising performance. This performance boost is only slightly more costly than a local exhaustive search, including the time spent building the tree, thanks to a simple parallelization.

Finally, we focused on how to obtain real-time performance with patch-based methods: First by proposing novel recursive algorithms, then GPU implementations of existing methods. A global patch search such as the one presented in Chapter 2 is still too costly for practical video denoising applications. This is why we presented a novel patch-based video denoising in Chapter 3 combining temporal filtering and non-locality. The temporal filtering is carried out by Kalman filters whose parameters are estimated non-locally. The resulting method is recursive: to denoise one frame, it uses solely information from the previous frame. Results show a performance comparable to more complex patch-based methods that use around ten frames to denoise each single frame. The main limitation of the method is that it relies heavily on the optical flow. Slight errors in registration cause a drop in PSNR. Nevertheless, visual quality of the result is superior, both in terms of details and temporal consistency, to algorithms with comparable running times.

In Chapter 4, we proposed a GPU implementation of NL-means, BM3D and VBM3D to further reduce computation times. This chapter focuses on the technical challenges of such implementations and how to overcome them. We also proposed compromises for the default parameters to get an even larger speedup with only a minor penalty for the denoising performance. This work paves the way to new uses of these algorithms in time constrained environments and for practical applications.

During the course of this thesis, the image processing community experienced a revolution as it was demonstrated that convolutional neural networks, trained with supervision, achieved much better performance than traditional methods across a wide range of image restoration tasks, including image denoising. This lead us to shift the focus of our research on how to apply those machine learning methods to video denoising. In particular, we described an effective way of incorporating temporal non-local information into a CNN for video denoising in Chapter 5. The proposed method computes for each image patch the n most similar neighbors on a spatio-temporal window and gathers the value of the central pixel of each similar patch to form a non-local feature vector which is given to a CNN. Our method yields a significant gain compared to other CNN approaches. It even has similar performance to the best non-CNN method evaluated, even outperforming them on the largest of our test datasets.

With the framework presented in Chapter 6, we show that a single video is often enough to "train" a denoising network, removing the need for a dataset of images. By applying a simple frame-to-frame training on a generic pre-trained network (for example a DnCNN network trained for additive Gaussian noise with fixed standard deviation), we successfully denoised a wide range of different noise models even though the network has never seen the video nor the noise model before its fine-tuning. This opens the possibility to easily process data from any unknown origin. We think that the current fine-tuning process can still be improved. First, given that the application is video denoising, it is expected that better results will be achieved by a video denoising network (the DnCNN network processes each frame independent of the others). Using the temporal information could improve the denoising quality, just like video denoising methods improve over frame-by-frame image denoising methods, but also might stabilize the variance of the result for the on-line fine-tuning. Recent works have also shown the possibility of real time fine-tuning [TRJ + 19, TTP + 19] paving the way to improve computation time.

Fine-tuning on a single video is very useful for data that has been processed by an unknown pipeline but it can also be used on raw data, i.e. mosaicked raw video. Indeed, it is possible to both demosaick and denoise at the same time with a single network. While videos are rarely shot in raw, this approach is particularly useful for image bursts that are more and more popular, especially with mobile phones. We have seen that the CNN based demosaicking methods beat by almost two decibels the best human-crafted methods while being faster by one order of magnitude in Chapter 7. To reach this performance, they did not rely on the clever human techniques established by the anterior state of the art, but simply applied rather standard CNN architectures. This success is explainable. The first reason is that human-crafted algorithms rely on the iteration of nonlinear local filters, the most complex one, ARI, having more than 20 such iterations. But deep convolution networks have the same structure and are in addition scalable in depth and number of filters, until they reach the best performance. Furthermore, human-crafted methods have been designed to avoid certain artifacts, probably to the cost of losing in PSNR. CNNs, on the contrary, can be taught to learn intricate casuistry by the variety of presented images, and to keep a memory of complex image statistics. Being trained to reach the best PSNR, they definitely reach that goal. Finally, the Gharbi et al. method succeeded in biasing the learning process so as to avoid completely the most annoying moiré and zipper effects, thus also achieving the most acute requirement of demosaicking methods, while still retaining the best PNSR. This performance arguably seals the destiny of human-crafted methods on this subject.

In Chapter 8, we proposed a novel way of training demosaicking neural networks without any RGB ground truth. We use instead other mosaicked data of the same scene (such as from a burst of images). Based on it and on recent neural network advances, we proposed a method to train jointly demosaicking and denoising with bursts of noisy raw images. We showed that finetuning on a given burst boosts the reconstruction performance. Clipped noise, a hard problem, is handled natively. It also presents a specific case where overfitting a network to the training data is valuable. Since we do not expect generalization there's only benefits from this overfitting. While the process of fine-tuning is costly and might not always be useful, training without ground truth is always an improvement to regular training. We hope our work can lead to new camera pipeline calibration procedures, and general improvement of the image quality when a burst is available.

The last part tackled a different set of problems, namely detection. In particular, we looked at anomaly detection in images and forgery detection and how these problems can be linked to denoising. We have first shown in Chapter 9 that anomalies are easier detected on a residual image, computed by removing the self-similar component, and then performing hypothesis testing. It is reassuring to see that our method finds all anomalies proposed in the literature with very low NFA. In addition, we have experimentally shown that the method verifies the non-accidentalness principle: no anomalies are detected in white noise. We plan to extend the method to videos, by analyzing anomalies in the motion field.

We then presented a theoretical analysis of the convergence of PatchMatch algorithms in Chapter 10. For an energy level ε, we show that the probability of having an energy above ε converges to zero with the number of iterations, and we provide worst case bounds on the convergence rate. Our analysis applies to the case of k nearest neighbors, and to most variants of PatchMatch proposed in the literature. We give specific bounds for two of these algorithms: the original PatchMatch [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF][START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF] and CHS [START_REF] Korman | Coherency sensitive hashing[END_REF]. For the case of the original PatchMatch (with k = 1) we validate our results by comparing the predicted convergence rate with the one found in practice. The setting of our framework is rather general: the task of patch matching is viewed as an optimization problem where the goal is to minimize several non-convex energies U x over the same domain. This might allow the application of these techniques to similar optimizations problems in other areas. For the case of matching patches, it would interesting to precise the link between the regularity of the images and the convergence rate, at least for certain simple models of images.

The forgery detection method analyzed in Chapter 11 takes advantage of PatchMatch. One of its major improvements is its resilience to rotations. It is also robust to image perturbations such as the addition of noise. Nevertheless, it is not deprived of drawbacks. In particular, the results are particularly disappointing when considering multiple copies of an object or when the forgery has been performed by a classic copy-paste method such as Poisson editing. A better choice for the descriptor should be able to fix this deficiency of the method though. Dense SIFT descriptors were considered as an alternative to improve the detection of forgeries performed by Poisson editing. Even though these descriptors allow for better detection, they are more likely to produce false positives. Nevertheless, this shows that the method can be modified and improved by varying the descriptors. The only drawbacks found -which are mainly linked to the usage of PatchMatch -is the inability of the method to detect multiple copies but also to not distinguish well similar objects from actual copies.

Finally, in Chapter 12 we have presented an unsupervised method to detect copy-move forgeries that is not only invariant to rotation, scaling and global change of illumination, but also robust to the presence of similar but genuinely different objects or regions. The method, being parameter-less and very fast, can be included in the necessary long series of tampering tests applied to a suspicious image. The limits of the method are closely linked to its strength. Because it is robust to the presence of naturally similar objects, it is less reliable in case of large degradations. We nevertheless found that the method is robust enough to usual noise and compression levels. An image that has been degraded too much is suspicious anyway since nowadays the quality of an image taken with a mobile is very good. Thus only images with a good enough quality should be tested. Highly degraded images would be anyway suspicious regardless of a sophisticated examination. The second limit is the usage of sparse keypoints. These keypoints are only computed in regions that are contrasted enough (non-flat areas) which means that forgeries in these regions might not detected. Finally matching keypoints give anchor points and do not delimit forged regions precisely. A natural extension of the method would be to extract the forged regions from the anchor points while still keeping a good control over the number of false detections. Finally coupling the method with a noise estimator could arguably make it still more discriminant. Proof. We will show the convergence in the mean for a single node x ∈ A, i.e. E[|U x (ϕ n x )|] ---→ n→∞ 0. We recall that for a non-negative random variable X, E[X] = x>0 P(X x). Then we have: Since Q 1 (η, ξ) > 0, for all η, ξ, we have that Q 1 (η, {U x = 0}) > 0 for any η. In the discrete case (the one only one considered for this theorem) the sup is achieved and is not 1. Therefore sup α>0 C 1 (x, α) = max α>0 C 1 (x, α) < 1 and the convergence follows. With a similar derivation can be used to show the convergence of the L ∞ norm of the whole NNF. For α > 0 we can guarantee that C (z, α) < 1. In the first part we focus on patch-based video denoising methods. We study in details VBM3D, a popular video denoising method, to understand the mechanisms that made its success. We also present a real-time implementation on GPU of this method.

E[|U x (ϕ n+1 x )|] = E[U x (ϕ n+1 x )] = ε>0 P(U x (ϕ n+1 x ) ε) (A.24)

A.3 Specific

We then study the impact of patch search in video denoising and in particular how searching for similar patches in the entire video, a global patch search, improves the denoising quality. Finally, we propose a novel causal and recursive method called NL-Kalman that produces very good temporal consistency. In the second part, we look at the newer trend of deep learning for image and video denoising. We present one of the first neural network architecture, using temporal self-similarity, competitive with state-of-the-art patchbased video denoising methods. We also show that deep learning offers new opportunities. In particular, it allows for denoising without knowing the noise model. We propose a framework that allows denoising of vi-deos that have been through an unknown processing pipeline. We then look at the case of mosaicked data.

In particular, we show that deep learning is undeniably superior to previous approaches for demosaicking. We also propose a novel training process for demosaicking without ground-truth based on multiple raw acquisition. This allows training for real case applications. In the third part we present different applications taking advantage of mechanisms similar those studied for denoising. The first problem studied is anomaly detection. We show that this problem can be reduced to detecting anomalies in noise. We also look at forgery detection and in particular copy-paste forgeries. Just like for patch-based denoising, solving this problem requires searching for similar patches. For that, we do an in-depth study of PatchMatch and see how it can be used for detecting forgeries. We also present an efficient method based on sparse patch matching.

Universit é Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France

Figure 2 :

 2 Figure 2: Scheme of the core of the VBM3D algorithm.

Figure 3 :

 3 Figure 3: The plots show the position in the spatio-temporal video domain of the matches found for a sample patch query for different search methods. From left to right: the best matches found with a global exhaustive search, with a local exhaustive search in a window centered at the query, and with the VPLR search, the heuristic proposed in this chapter. Notice how the latter discovers the patch trajectories similar to those of the global exhaustive search.

Figure 5 :

 5 Figure 5: The architecture of the proposed method. The first module performs a patch-wise nearest neighbor search across neighboring frames. Then, the current frame, and the feature vectors f nl of each pixel (the center pixels of the nearest neighbors) are fed into the network. The first four layers of the network perform 1 × 1 convolutions with 32 feature maps. The resulting feature maps are the input of a simplified DnCNN [ZZC + 17a] network with 15 layers.

Figure 6 :

 6 Figure 6: From the same starting point and only using the video, our fine-tuned network is able to denoise different noises without any artifact. The top images are the noisy and the bottom ones the denoised. From left to right: Gaussian noise, Poisson type noise, salt and pepper type noise and JPEG compressed Gaussian noise.

Figure 7 :

 7 Figure 7: From top-left to bottom-right: Original, Gharbi et al. [GCPD16], Tan et al. [TZZZ17], MLRI [KMTO14], ARI [MKTO15], Getreuer [Get12]. Learning based methods produce fewer artifacts while still being very efficient computationally.

Figure 10 :

 10 Figure 10: Detection of forgery using PatchMatch and Zernike moments for a forgery applying only a translation.

•

  Global patch search boosts video denoising, Thibaud Ehret, Pablo Arias and Jean-Michel Morel, International Conference on Computer Vision Theory and Applications, 2017 • Automatic Detection of Internal Copy-Move Forgeries in Images, Thibaud Ehret, Image Processing On Line, 2018 • Model-blind video denoising via frame-to-frame training, Thibaud Ehret, Axel Davy, Jean-Michel Morel, Gabriele Facciolo and Pablo Arias, Conference on Computer Vision and Pattern Recognition, IEEE, 2018 • Non-Local Kalman: A recursive video denoising algorithm, Thibaud Ehret, Jean-Michel Morel and Pablo Arias, International Conference on Image Processing, IEEE, 2018

Figure 11 :

 11 Figure11: On the left image the two objects are similar but not digital copies of each other. On the right, one is a digital copy of the other. The patches shown below each respective image correspond to the red squares in the images. They show that a difference is visible at this level and therefore the descriptors can be discriminated. This is why the detection method can discard genuinely similar objects.

Figure 12 :

 12 Figure 12: Bien que l'image produite par un téléphone Samsung Galaxy S7 ne semble pas être bruitée (gauche), l'image brute correspondante telle que vue par le capteur est en fait très bruitée (droite). L'image finale est produite par une chaîne complexe de traitement d'image qui applique un débruitage lors d'une des différentes étapes.

Figure 13 :

 13 Figure 13: Schéma expliquant le principe de fonctionnement de VBM3D.

Figure 14 :

 14 Figure 14: Ces images représentent la position des meilleurs patches dans la vidéo trouvés par différentes techniques de recherche de patches. De gauche à droite: la position des meilleures correspondances pour la recherche exhaustive globale, pour la recherche exhaustive locale centrée sur la position du patche de référence et l'heuristique proposée dans ce chapitre. Il est important de remarquer que cette dernière trouve des trajectoires de patches semblables aux trajectoires trouvées par la recherche exhaustive globale.

Figure 15 :

 15 Figure 15: Résultats de débruitage de séquences corrompues par un bruit gaussien avec un écart-type de 30. De gauche à droite: Débruitage avec NL-Kalman, NL-Kalman avec un flot optique oracle, VBM3D, VBM4D et l'image de référence. De haut en bas: Extrait de la séquence crowd_run et pedestrian_area. La méthode permet de récupérer plus de détails comme par exemple dans l'arbre ou dans le texte.

Figure 16 :

 16 Figure 16: L'architecture du réseau de neurones proposé. Le premier module effectue une recherche de patches similaires à travers différentes images vidéo. L'image vidéo de référence ainsi que l'information de similarité (donnée sous la forme d'un vecteur des pixels centraux des patches trouvés) sont données au réseau. Les quatre premières couches du réseau sont des séries de 32 convolutions de taille 1 × 1. Le résultat de ces convolutions est ensuite l'entrée d'un réseau de type DnCNN [ZZC + 17a] simplifié avec 15 couches.

Figure 17 :

 17 Figure 17: À partir du même point de départ et en utilisant uniquement la vidéo disponible, notre méthode de raffinage de réseau de neurones est capable d'apprendre à débruiter différents types de bruit sans produire d'artefacts. Les images du haut sont les images bruitées et celles du bas les débruitées. De gauche à droite: bruit gaussien, poisson, poivre et sel et bruit gaussien compressé par JPEG.

Figure 18 :Figure 19 :

 1819 Figure 18: De haut en bas et de gauche à droite: Référence, Gharbi et al. [GCPD16], Tan et al. [TZZZ17], MLRI [KMTO14], ARI [MKTO15], Getreuer [Get12]. Les méthodes basées sur l'apprentissage produisent moins d'artefacts tout en ayant de bons temps de calculs.

Figure 20 :

 20 Figure 20: Exemples de détections pour notre méthode en utilisant soit directement les pixels de l'image, soit en utilisant les cartes d'activations du réseau de neurones VGG pour différentes couches (conv1_1, conv2_-1, conv3_1) avec une comparaison à [MC13], [IKN98], [JWY + 13] et [HSBZ15] sur des exemples réels. Les deux premières images (première et deuxième lignes) font parties de la base de donnée Toronto [BT06], les exemples de la troisième et quatrième lignes proviennent de [MC13] et [TH99] respectivement.

  Figure 21: Détection de falsification en utilisant PatchMatch et des moments de Zernike pour une falsification de type copié-collé avec une translation.

Figure 22 :

 22 Figure 22: À gauche, une image avec deux objets similaires mais qui ne sont pas des copies. À droite, une image où les objets sont des copies. Les patches montrées en dessous de chaque image correspondent aux carrés rouges. Ces patches montrent qu'une différence est visible à cette échelle et que les descripteurs peuvent être différenciés. C'est pour cela que la méthode peut discerner entre des copies ou juste des objets similaires.

Figure 1

 1 Figure 1.1: Scheme of the core of the VBM3D algorithm.

Figure 1 . 2 :

 12 Figure 1.2: The patch search starts with a local spatial search in the current frame of size Ns × Ns. Only the N b best candidates are then used in the following frame to predict where to search. This leads to N b local spatial searches (each centered at the candidates from the previous frame), of size Npr × Npr. In practice, Ns = 7, Npr = 5 and N b = 2.

Figure 1

 1 Figure 1.3: Top: results of VBM3D [DFE07], with the original authors' implementation. Bottom: result obtained with our implementation. The noise level is σ = 40. The contrast has been linearly scaled for better visualization.

Figure 1 . 4 :

 14 Figure 1.4: From left to right: result of VBM3D (our implementation with default parameters), VBM3D ST, VBM3D OF, VBM3D ST+OF, VBM3D ST+OF+MS (Lanczos multiscaler) and the original clean video. The noise level is σ = 40. Contrast has been linearly scaled for better visualization.

Figure 2 . 1 :

 21 Figure 2.1: The plots show the position in the spatio-temporal video domain of the matches found for a sample patch query for different search methods. From left to right: the best matches found with a global exhaustive search, with a local exhaustive search in a window centered at the query, and with the VPLR search, the heuristic proposed in this chapter. Notice how the latter discovers the patch trajectories similar to those of the global exhaustive search.

Figure 2 . 3 :Figure 2 . 4 :

 2324 Figure 2.3: Comparison of the PSNR frame by frame for different methods on the grayscale bus sequence with noise 10.

Figure 2 . 5 :

 25 Figure 2.5: Results of denoising for mobile (zoom on frame 37, noise 20). Top: ground truth, NL-Bayes local, BM3D local, VBM4D; bottom: Noisy, NL-Bayes global and BM3D mix.

Figure 2 . 6 :

 26 Figure 2.6: Results of denoising for grayscale bus (zoom on frame 70, noise 20). Top: ground truth, NL-Bayes local, BM3D local, VBM4D; bottom: Noisy, NL-Bayes global, BM3D mix and VBM3D.

Figure 3 . 1 :

 31 Figure 3.1: Evolution of the quality of the denoising during the warm-up stage. Artifacts due to the spatial denoising quickly disappear after few iterations of the temporal filtering.

Figure 3 . 2 :

 32 Figure 3.2: Denoising results when the sequence is corrupted by noise of standard deviation 30. From left to right: Denoising using NL-Kalman, NL-Kalman with the oracle optical flow, VBM3D, VBM4D and the original.From top to bottom: Crop from the sequence crowd_run and pedestrian_area. Overall the proposed method preserves more details as can be seen both in the tree and in the text. Results best viewed zoomed.

Original

Figure 4 6 Aggregate all pixels of p in v 7 else 8 = N i=1 w i p i 10 Aggregate

 46810 Figure 4.1: Scheme of the NL-means algorithm

Figure 4 . 2 :

 42 Figure 4.2: Scheme of the BM3D algorithm

Algorithm 12 :

 12 Keeping an unordered table of distances and positionsInput: New position pos and distance dist, two tables Positions and Distances of length n each.1 max_distance ← max(Distances[0], . . . , Distances[n-1]) 2 if dist < max_distance then 3 for i from 0 to n-1 do 4 if Distances[i] = max_distance then 5 Positions[i] ← pos 6 Distances[i] ← dist 7 quit functionpiler reorganizing the iterations in an unoptimized way for this algorithm, while not having issues with Algorithm 12.

Figure 4 . 3 :

 43 Figure 4.3: Memory layout of NL-means. Buffers in green are read-only and buffers in blue are read-write. Note that there is no intermediate buffer for the processing of the patches.

  4 for k from 0 to n-1 do 5 data ← u[k-th neighbor row i col j] // for patches of width p, p consecutive threads access the same patch at consecutive columns 6 mean ← mean + data 7 var ← var + data * data 8 mean ← reduce_add(mean) / (p*p*n) 9 var ← reduce_add(var) / (p*p*n) // The reduce operation is only among the p threads treating a same patch var ← var -mean * mean flatpatch ← var <= βσ 2 for i from 0 to p-1 do if flatpatch then denoised_pixel ← mean else denoised_pixel ← 0 for k from 0 to n-1 do 18 denoised_pixel ← denoised_pixel + w[k-th neighbor] * u[k-th neighbor row i col j]

Algorithm 14 :

 14 Emulated atomic float add Input: Write address p and content to add a 1 current ← p[0] // Reads initial value 2 repeat 3 expected_current ← current 4 next ← expected_current + a // Compute float add 5 current ← atomic_cmpxchg(p, expected_current, next) // Try to update 6 until as_uint(current) = as_uint(expected_current) // Repeat if value changed before updating presented on Algorithm 14.

Figure 4 . 4 :

 44 Figure 4.4: Memory layout of BM3D and VBM3D. Buffers in green are read-only and buffers in blue are read-write. Note that there is no intermediate buffer for the processing of the patch stack.

  17 × 17 patches, but with a heavy architecture. More recently, DnCNN [ZZC + 17b] obtained impressive results with a far lighter 17 layer deep CNN with 3 × 3 convolutions, ReLU activations and batch normalization [IS15]. This work also proposes a blind denoising network that can denoise an image with an unknown noise level σ ∈ [0, 55], and a multi-noise network trained to denoise blindly three types of noise. A faster version of DnCNN, named FFDNet, was proposed in [ZZZ18], which also allows handling spatially varying noise by adding a noise map σ(x) as an additional input. The architectures of DnCNN and FFDNet keep the same image size throughout the network. Other architectures [MSY16, SMD17, CCXK18] use pooling or strided convolutions to downscale the image, and then up-convolutional layers to upscale it back. Skip connections connect the layers before the pooling with the output of the up-convolution to avoid loss of spatial resolution. Skip connections are used extensively in [TYLX17].noisy frame + n-1 non-local features:

Figure 5 . 1 :

 51 Figure 5.1: Illustration of the proposed Video Non-Local Network (VNLnet). The first module performs a patch-wise nearest neighbor search across neighboring frames. Then, the current frame, and the feature vectors f nl of each pixel (the center pixels of the nearest neighbors) are fed into the network.

Figure 5

 5 Figure 5.2: Spatio-temporal video crop (top row) and the features extracted by the non-local search for the central frame (bottom row), applying the restriction of one neighbor per frame.

Figure 5 . 3 :

 53 Figure 5.3: Results on still image denoising (AWGN with σ = 25). Original clean image, noisy image, result obtained with the baseline CNN, result of incorporating the non-local features by finding the nearest neighbors on the noisy image or the oracle noise-less image. Contrast has been linearly scaled for better visualization.

  Figure 5.4: Example of denoised results with our method when changing the patch size, respectively no patch search, 9 × 9, 15 × 15, 21 × 21, 31 × 31 and 41 × 41 patches. The 3D search window has 15 frames for these experiments.

Figure 5 . 5 :

 55 Figure 5.5: Example of denoised results with our method when changing the number of frames considered in the 3D search window (respectively no patch search, 3, 7, 11 and 15). 41 × 41 patches were used for these experiments.

Figure 5 . 6 :

 56 Figure 5.6: Example of denoised result for Non-Local Pixel Mean, DnCNN, video-DnCNN (see Section 5.4.3) and VNLnet, for AWGN with σ = 20. The four crops highlight the results on frames feature various kinds of motion. The videos are part of the DAVIS dataset. Non-Local Pixel Mean corresponds to the average of the output of the non-local patch search.

Figure 5

 5 Figure 5.7: PSNR gain of VNLnet versus the network without patch search of Table 5.3 (15 input frames) on grayscale DAVIS (σ = 20) (35.48db versus 34.59db).

  gain given patch texturedness same, but weighted by the frequency of the texturedness patch gradient magnitude (dB) given patch texturedness same, but weighted by the frequency of the texturedness

Figure 5

 5 Figure 5.8: Patch PSNR gain between VNLnet versus video-DnCNN (top) and DnCNN (bottom) on grayscale DAVIS with σ = 20, as a function of a measure of the patch texturedness (defined here by the patch gradient magnitude). The grayscale image depicts the 2D histogram of PSNR gains and patch texturedness computed from all 41 × 41 patches in the test set. The red curve shows the average PSNR gain for each texturedness level, and the blue curve shows the same average PSNR gain, but weighted by the frequency of each texturedness level.

Figure 5 . 9 :

 59 Figure 5.9: Example of denoised result for several algorithms (AWGN with σ = 20) on a sequence of the color DAVIS dataset [PTPC + 17]. The crops highlight the results on non-moving and moving parts of the video.

Figure 6 . 1 :

 61 Figure 6.1: From the same starting point and only using the video, our fine-tuned network is able to denoise different noises without any artifact. The top images are the noisy and the bottom ones the denoised. From left to right: Gaussian noise, Poisson type noise, salt and pepper type noise and JPEG compressed Gaussian noise.

Figure 6 . 2 :

 62 Figure6.2: The fine-tuning process is done on a sequence corrupted by an additive Gaussian noise of standard deviation σ = 25 (left) or σ = 50 (right). The fine-tuned networks (batch and online) achieve comparable performance than the reference networks.

  Figure 6.3: Different types of noise. From top-left to bottom right: multiplicative Gaussian noise, correlated Gaussian noise, salt and pepper noise, Gaussian noise after JPEG compression. The fine-tuned network (both online and batch) always performs better than the original network.

Figure 6 . 4 :

 64 Figure 6.4: Example of denoising of an image corrupted by a JPEG compressed Gaussian noise. The finetuned network doesn't produce any visible artifacts, contrary to the original DnCNN used for the fine-tuning process. From left to right, top to bottom: Noisy, fine-tuned, VBM3D, ground truth, DnCNN trained for a Gaussian noise, noise clinic.

Figure 6 . 5 :

 65 Figure 6.5: Example of denoising of an image corrupted by a Gaussian noise of standard deviation σ = 50. The fine-tuned network doesn't produce any visible artifact, the results are comparable to a DnCNN trained for this particular type of noise. From left to right, top to bottom: Noisy, fine-tuned, DnCNN trained for a Gaussian noise with σ = 50, VBM3D, ground truth, noise clinic, DnCNN trained for a Gaussian noise with σ = 25.

Figure 6 . 6 :

 66 Figure6.6: Impact of parameters. Left: Impact of the learning rate and the number of iterations. It also shows the gap between using a pre-trained network and random initialization. Right: Impact of the number of frames used for fine-tuning.

Figure 6

 6 Figure 6.7: Lifelong learning. Left: Slow change. Right: sudden change. The fine-tuned network adapts without difficulty to slow changes and sudden changes. See text for more details

Figure 6 . 8 :

 68 Figure 6.8: Blind denoising (on the right) of images coming from a video taken during World War I (original on the left). Blind denoising is useful in this case because it would be nearly impossible to recreate this type of noise to train a network.

Figure 6 . 9 :

 69 Figure 6.9: Blind denoising better preserves details than methods for a predefined noise. From top to bottom, left to right: original, blind denoising, denoising with the pre-trained network and VBM3D with handtuned noise parameter.

Figure 6 . 10 :

 610 Figure 6.10: Blind denoising (on the right) of images coming from a video taken during World War I (original on the left). Blind denoising is useful in this case because it would be nearly impossible to recreate this type of noise to train a network.

Figure 6 . 11 :

 611 Figure 6.11: Example of denoised image (right) coming from a mobile phone (left). The results is more natural and pleasing to the eye as it doesn't have all these ugly artifacts.

Figure 7

 7 Figure 7.1: Architecture used for demosaicking by Gharbi et al. [GCPD16].

Figure 7 . 3 :

 73 Figure 7.3: Example of images from the luminance dataset (two on the left) and the moiré dataset (two on the right).

Figure 7 . 4 :

 74 Figure 7.4: Architecture for demosaicking used by Tan et al. [TZZZ17].

Figure 7 . 5 :

 75 Figure 7.5: Example of images from the Waterloo dataset [MDW+ 17]

Figure 7 . 6 :

 76 Figure 7.6: From top-left to bottom-right: Original, Gharbi et al. [GCPD16], Tan et al. [TZZZ17], MLRI [KMTO14], ARI [MKTO15], Getreuer [Get12]. All demosaicking methods shown here perform well overall, with no visible strong artifacts. Yet a moiré artifact appears in the fence for some of the methods.

Figure 7 . 7 :

 77 Figure 7.7: From top-left to bottom-right: Original, Gharbi et al. [GCPD16], Tan et al. [TZZZ17], MLRI [KMTO14], ARI [MKTO15], Getreuer[START_REF] Getreuer | Image Demosaicking with Contour Stencils[END_REF]. All demosaicking methods still present some moiré artifacts on the shutters. However, the Gharbi et al.method, which was trained specifically to avoid this artifact, has fewer than the other methods.

Figure 7

 7 Figure 7.8: From top-left to bottom-right: Original, Gharbi et al. [GCPD16], Tan et al. [TZZZ17], MLRI [KMTO14], ARI [MKTO15], Getreuer [Get12]. Most methods perform well regarding luminance artifacts. Nevertheless the Gharbi et al.method still performs best.

Figure 7 . 9 :

 79 Figure 7.9: From top-left to bottom-right: Original, Gharbi et al. [GCPD16], Tan et al. [TZZZ17], MLRI [KMTO14], ARI [MKTO15], Getreuer [Get12]. Only the Gharbi et al.method performs well visually on really difficult examples.

Figure 8 . 1 :

 81 Figure 8.1: Using a burst, our fine-tuning (starting from the network from Gharbi et al. [GCPD16]) is able to not only denoise well (σ = 5) but also doesn't show any artifacts like zipper or moire in the difficult regions. Best visualized on a screen.

Figure 8 . 2 :

 82 Figure 8.2: Proposed pipeline to train for demosaicking without using any ground truth. The output after applying the network D on the first image is warped using the transform T and masked with M so to be compared to the second masked mosaicked image. The black corners seen in at the last stage of the diagram indicate the undefined pixels after the transform, which are not considered by the loss.

Figure 8 . 3 :

 83 Figure 8.3: Architecture of the network used to compare the performance of learning on RGB ground truth or only with pairs of RAW images.

Figure 8 . 4 :

 84 Figure 8.4: Evolution of the average PSNR on the validation dataset when training with ground truth data and when training without RGB ground truth data available. Training without RGB ground truth behaves the same than training with an RGB ground truth.

Figure 8

 8 Figure8.5: Fine-tuning a denoising network (DnCNN [ZZC + 17a] σ = 25) on a burst of 10 noisy (σ = 25) grayscale images with saturated regions. From left to right: PSNRs over the whole image, on non-saturated regions, and on the saturated regions. After fine-tuning, the network works better both on saturated and non-saturated regions. While L2 is not able to deal with clipping, using L1 for fine-tuning performs similarly to fine-tuning without clipping.

Figure 8 . 6 :Figure 8

 868 Figure 8.6: From left to right: reference image, noisy (σ = 25), pretrained DnCNN and DnCNN after finetuning. The details, such as the trees, are sharper and more distinguishable after fine-tuning. Figure best visualized zoomed-in on a computer.

Figure 8

 8 Figure8.8: Fine-tuning a pre-trained denoising network (DnCNN σ = 25) to a specific sequence increase the quality of the result. The visible drops correspond to each change of image considered (pairs are considered in lexicographical order). It is important to finish with the reference image as to maximise the performance. The fine-tuned network, which takes as input a single frame, comes close to the performance of DnCNN applied to the temporal average of all frames.

Figure 8 . 9 :

 89 Figure 8.9: From left to right: reference image, the network from §8.2, the network from §8.2 after finetuning and Gharbi et al.. Because of the reduced size of the training set our blind network still has some moire artifact but they completely disappear after fine-tuning on the data achieving a result visually close to Gharbi et al.without having to learn on a specific well-chosen dataset. Figure best visualized zoomed-in on a computer.

Figure 8 . 10 :

 810 Figure8.10: Fine-tuning a pre-trained demosaicking network (from Section 8.2) to a specific sequence increase the quality of the result. The visible drops correspond to each change of image considered (pairs are considered in lexicographical order). It is important to finish with the reference image as to maximise the performance.

Figure 8 . 11 :

 811 Figure 8.11: Experiment on a real burst. Left to right: The result of the HDR+ pipeline [HSG + 16], bilinear interpolation, [GCPD16] and [GCPD16] with our fine-tuning. Contrast was enhanced for all methods except HDR+. Note that the HDR+ pipeline includes color balance as well as sharpening. It also uses all the images of the burst to produce the result (all other methods use only the reference frame). Figure best visualized zoomed in on a computer.

Figure 9

 9 Figure 9.1: Image anomalies are successfully detected by removing all self-similar content and then looking for structure in the residual noise, which parameters are easy to estimate. First row: an image with a color anomaly (the red dot); (a): the detections (first column) obtained from principal components of CNN feature maps (2nd-5th columns) (b): the detection map on the same features obtained after removing the self-similar content. In the detection maps, cyan means good detection and orange extremely salient detection.

  Figure9.2: Our method applies the same framework at all the different scales. The framework computes the difference between the image (possibly using a dense representation using a neural network) and its self-similar model before normalizing this difference output distribution. The detection is then done on these normalized distributions. The corresponding pseudocode is presented in 20.

  Figure 9.3: Steps of the VPLR search

  2 for all α in [0.5, 0.6, . . . , 1.4] do 3 Test if r α is Laplacian 4 Test if r α is Gaussian 5 The best distribution D and α are kept. // Best distribution in terms of minimizing the l 2 distance from the uniform distribution of the cumulative histogram of cdf D (r α ), where D is the tested distribution. 6 r ← inv_cdf Gaussian(0,1) (cdf D (r α )) // Applied element-wise 7 number of false alarms under the null-hypothesis, namely:

Figure 9

 9 Figure 9.5: A ground truth (on the right) for anomaly detection has been generated by introducing an anomaly in a RPN [GGM11b] texture (on the left), which is anomaly free. The detection is then done on the result (in the middle).

Figure 9

 9 Figure 9.7: ROC curve computed on the dataset of synthetic images. A true positive corresponds to an anomalous pixel detected. A false positive corresponds to a normal pixel that has been detected as anomalous. In deep blue [AT10] (Area Under the Curve (AUC) 7.52), in red [BCW14] (AUC 5.79), in yellow [DEMD18] (AUC 7.03), in purple [GM09] (AUC 3.30), in green [MC14] (AUC 1.98) and in light blue [ZC10] (AUC 2.92).

Figure 9

 9 Figure 9.8: ROC curve computed on the dataset of synthetic images. A true positive is when an anomaly is detected (in the sense that at least one detection has been made inside the anomalous region). A false positive is when there is a detection outside the anomalous region. In deep blue [AT10] (Area Under the Curve (AUC) 0.82), in red [BCW14] (AUC 0.585), in yellow [DEMD18] (AUC 0.87), in purple [GM09] (AUC 0.52), in green [MC14] (AUC 0.28) and in light blue [ZC10] (AUC 0.625).

Figure 9 . 10 :

 910 Figure 9.10: From left to right: image of an undersea mine from [MC14], image of a periodic textile from [TH99], image of a tumor from [GM09], image of a man from the Toronto dataset [BT06], image of a garage door from [BT06]. From top to bottom: The original image, the image residual of one of the scales computed in [DEMD18] (the scale shown is the one where the anomaly is the most salient, the contrast has been adjusted for visualization purpose), algorithm detections for: [DEMD18], [AT10], [ZC10], [MC14],[GM09] and[START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF]. Detections are shown using the following color coding: white is a weak detection -threshold with NFA ∈ [10 -3 , 10 -2 ], cyan is a mild detection -threshold with NFA ∈ [10 -8 , 10 -3 ], green is a strong detection -threshold with NFA ∈ [10 -21 , 10 -8 ], and orange is very strong -threshold with NFA ≤ 10 -21 . When available red is the detection with the threshold corresponding to the lowest NFA. For[START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] we adopted a similar color coding: white between 0 and 0.5, cyan between 0.5 and 0.7, green between 0.7 and 0.9 and orange above 0.9.

Definition 10 . 1 (

 101 Matching energy). The quality of the matching of a patch x ∈ A is measured by an the energy function U x (.) defined for any patch z ∈ R d with values between 0 and +∞. We generalize this definition to a set of patches u by U x (u) = max y∈u U x (y).(10.1)

  µ(x) = |{y | y ∼ x}|.(10.5)

  η∈{Uz ε} Q(η, {U z ε}).(10.8)Lemma 10.1. For all z ∈ A, C(z, .) is a non-increasing function such that for ε < 0, C(z, ε) ∈ [0, 1[ and C(z, ε) = 1 for ε 0.

Algorithm 28 :2

 28 Generic patch matching algorithm 1 Initialize propagation graph G Initialize matching ϕ 0 3 for n ∈ N do 4 Update candidates 5 for x ∈ V following the topological ordering do 6 Convergence of the patch matching algorithms

Figure 10 . 1 :Figure 10 . 2 :

 101102 Figure 10.1: We represent here the εz,xs for two different points x (top and bottom rows) and for ε = 1 (left) and ε = 15 (right). The point x is the bottom right pixel.

Figure 10 . 3 :

 103 Figure10.3: The effect of the random search. If we simplify the random search and make it uniform, then convergence is slower and the bound is tight. In this experiment, we match two random noise images. See text for details.

Figure 10 . 4 :

 104 Figure 10.4: Comparison between the "average" and worst-case C. The curve with the diamonds corresponds to the "average" C whereas the stars show the worst case C.

  Figure 11.1: Different types of samplings for the Zernike moments.

  Compute the coefficients for Equation (11.16) 4 for ρ = 0 to sp -1 do 5 for θ = 0 to 4(2ρ + 1) -1 do 6 foreach (n, m) s.t. n ∈ 1, o , m ∈ 1, n and n ≡ m[2] do 7 w ← 0 // Initialize the weights 8 for s = 0 to n-m ) -s x h ρ sin 2θπ 4(2ρ+1) -s y w + F nm sxsy See Equation (11.18) 18 foreach filter F nm computed previously doConvolve I (with a null extension) with F nm and store its modulus in ZM nm .

  Figure 11.3: Different types of PatchMatch propagations.

. 20 )

 20 This approximation is classic in numerical analysis. Translating Equation(11.20) to the current problem, A being any composition of L, R, U and D, leads tos(a) ≈ 2s(A(a)) -s(A • A(a)).(11.21)The first-order nodes used for these types of propagations are represented in red in Figure11.3b.In the end, a total of eight propagations of the different types is done during every iteration of PatchMatch.Adding all these modifications to the original PatchMatch algorithm presented in Chapter 10 gives us the version used to solve the problem at hand. The final modified PatchMatch algorithm is summarized in Algorithm 31.

  Algorithm 31: medianFilter algorithm input : A displacement map D, the radius of the filter ρ m output : A filtered displacement map D 1 foreach pixel p in D do 2 D(p) ← median({D(p ) | p ∈ D(p, ρ m )})

  ∆ be the displacement of these same points (still in homogeneous coordinates). Let A ∈ M 3×3 (R) be a matrix representing an affine transform. We are looking for a region such that the transformation error defined by(P ) = min A∈M 3×3 (R) ∆ -(AP -P ) 2 , (11.23)is small. The best affine transform minimizing Equation (11.23) can be estimated using a simple Least Square Regression, i.e.A = arg minA ∈M 3×3 (R) ∆ -(A P -P ) 2 . (11.24)The solution of Equation (11.24) can be expressed asA = (P + ∆)P T (P P T ) -1 , (11.25) therefore the error can be estimated without a minimization step by replacing A by its solution from Equation (11.25) (P ) = ∆ -(AP -P ) 2= ∆ -((P + ∆)P T (P P T ) -1 P -P ) 2

  ) 3 H ← I -H 4 foreach pixel p in D do 5 Let ∆ be the displacements corresponding to D(p, ρ e ) 6 e ← ∆H 2 // See Equation (11.26) 7 E(p) ← 1 if e < τ 0 otherwise Half-size of patch used to compute the moments sp 8 Number of PatchMatch iterations N 8 Minimum distance between two PatchMatch matches τ m 8 Minimum distance between two clones τ d 50 Error threshold τ 300 Minimum size of a clone τ s 1200 Radius of the median filter ρ m 4 Radius of the error filter ρ e 6 Table 11.1: Value of the parameters used during the computation of the results in Section 11.5.1.

Figure 11 . 4 :

 114 Figure 11.4: Color scheme representing the direction and magnitude of the displacement

Figure 11 . 5 :

 115 Figure 11.5: Result of the method using Zernike moments on (from top to bottom): a forged image with a forgery applying only a translation, a clean image, a forged image with a forgery applying a translation and a ten degrees rotation (resampled Zernike moments), a forged image with a forgery applying a translation and a ten degrees rotation (basic Cartesian Zernike moments).

Figure 11 . 6 :

 116 Figure 11.6: Result of the method using Zernike moments on (from top to bottom): a forged image with a forgery applying a translation on top of which a noise of standard deviation 25 was added, a forged image with a flipped copy, a forged image for a forgery applying a Poisson editing, a forged image for a forgery applying translations when a same object is copied at multiple different positions.

Minimum distance between two clones τ d 50 Error threshold τ 300 Minimum size of a clone τ s 1200 Radius of the median filter ρ m 4 Radius of the error filter ρ e 6 Figure 11 . 7 :Figure 11

 30012004611711 Figure 11.7: Result of the method using dense SIFT on (from top to bottom): a forged image with a forgery applying only a translation, a forged image with a forgery applying a translation and a ten degrees rotation, a clean image.

Figure 12

 12 Figure12.1: On the left image the two objects are similar but not digital copies of each other. On the right, one is a digital copy of the other. The patches shown below each respective image correspond to the red dots in the images. They show that a difference is visible at this level and therefore the descriptors can be discriminated. This is why the detection method can discard genuinely similar objects.

τ

  . For the distance d max defined byd max (D i , D j ) = max k,l∈{1,...,N } test is simply d max (D i , D j ) τ .This corresponds to a stricter version of matching than the one presented to match patches in Chapter 3.

Figure 12

 12 Figure 12.2: Examples of forgeries that were successfully detected in the different datasets. From the top to bottom, left to right: an example with a rotation, a change of illumination, a change of scale and with JPEG compression.

Figure 12 . 3 :

 123 Figure 12.3: Examples of forgeries that were not successfully detect in the different datasets. From top to bottom, left to right: an example where the forged region is completely flat, with a non-uniform change of illumination, with a tilt applied and the false detection.

Theorem 11 . 2 .

 112 Consider the field of candidate matches at iteration n, ϕ n . Define ϕ n+1 by applying an iteration of the Generic PatchMatch in Algorithm 28. Then, for all ε > 0 we haveP( U . (ϕ n+1 . ) ∞ ε) P( U . (ϕ n . ) ∞ ε) z∈A C 2 (z, ε) µ(z) C 1 (z, ε) . (A.23)Proof. The proof is similar to the one of Theorem 11.1 where instead of using Lemma 11.2 we use U . (ϕ n+1 . ) ∞ ε, i.e. ∀x ∈ A, U x (ϕ n+1 x ) ε. Corollary 11.1. Assume that for any pair (η, ξ) of sets of k candidate matches Q 1 (η, ξ) > 0 (or Q 2 (η, ξ) > 0). Let (ϕ n ) be a sequence defined by Algorithm 28. Then ∀x ∈ A, E[U x (ϕ n x )] ---→ n→∞ 0 and E[ U . (ϕ n . ) ∞ ] ---→ n→∞ 0.

C 1 C 1

 11 ε>0 z∈A C 2 (z, ε z,x ) µ(z) C 1 (z, ε z,x ) P(U x (ϕ n x ) ε) (A.25) ε>0 C 2 (x, ε) µ(x) C 1 (x, ε) P(U x (ϕ n x ) ε) (A.26) (x, α) P(U x (ϕ n x ) ε) (x, α) E[U x (ϕ n x )].(A.29)

  PatchMatch algorithms A.3.1 The original PatchMatch algorithm Proposition 11.1. The specific basic PatchMatch algorithm described in this section algorithm converges in probability to a NNF which minimizes the energy, namelylim n→∞ P(U x (ϕ n x ) ε) = 0, ∀ε > 0, x ∈ A, (A.30)with a geometric convergence rate. Moreover for all ε > 0, for all x ∈ A, we have thatP(U x (ϕ n+1 x ) ε) P(U x (ϕ n x ) ε) z∈A 1 -1 -C (c i , ε z,x ) k , (A.31) with C (z, α) := sup η Q (η, {U z a}). (A.32)

Titre:

  D ébruitage vid éo et applications Mots cl és : D ébruitage, D ébruitage à patches, Apprentissage profond, D émosaïquage, D étection d'anomalies, PatchMatch, D étection de falsification R ésum é : Cette th èse est d édi ée au d ébruitage vid éo. La premi ère partie se concentre sur les m éthodes à patches pour le d ébruitage de vid éo. Nous étudions en d étail VBM3D, une m éthode populaire de d ébruitage vid éo, pour comprendre les m écanismes qui ont fait son succ ès. Nous pr ésentons aussi une impl émentation temps-r éel sur carte graphique de cette m éthode. Nous étudions ensuite l'impacte de la recherche de patches pour le d ébruitage vid éo et en particulier comment une recherche globale peut am éliorer la qualit é du d ébruitage. Enfin, nous proposons une nouvelle m éthode causale et r écursive appel ée NL-Kalman qui produit une tr ès bonne consistance temporelle. Dans la deuxi ème partie, nous étudions les m éthodes d'apprentissage pour le d ébruitage. Nous pr ésentons l'une des toutes premi ères architecture de r éseau qui est comp étitive avec l' état de l'art. Nous montrons aussi que les m éthodes bas ées sur l'apprentissage profond offrent de nouvelles opportunit és. En particulier, il devient possible de d ébruiter sans connaître le mod èle du bruit. Gr âce à la m éthode propos ée, m ême les vid éos trait ées par une chaîne de traitement inconnue peuvent être d ébruit ées. Nous étudions aussi le cas de donn ées mosaïqu ées. En particulier, nous montrons que les r éseaux de neurones sont largement sup érieurs aux m éthodes pr éc édentes. Nous proposons aussi une nouvelle m éthode d'apprentissage pour le d émosaïquage sans avoir besoin de v érit é terrain. Dans une troisi ème partie nous pr ésentons diff érentes applications aux techniques utilis ées pour le d ébruitage. Le premier probl ème étudi é est la d étection d'anomalie. Nous montrons que ce probl ème peut être ramen é à d étecter des anomalies dans du bruit. Nous regardons aussi la d étection de falsifications et en particulier la d étection de copi é-coll é. Tout comme le d ébruitage à patches, ce probl ème peut être r ésolu à l'aide d'une recherche de patches similaires. Pour cela, nous étudions en d étail PatchMatch et l'utilisons pour d étecter des falsifications. Nous pr ésentons aussi une m éthode bas ée sur une association de patches parcimonieuse. Title : Video denoising and applications Keywords : Denoising, Patch-based denoising, Deep learning, Demosaicking, Anomaly detection, Patch-Match, Forgery detection Abstract : This thesis studies the problem of video denoising.

  Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Copy-move matching with SIFT-like matching . . . . . . . . . . . . . . . . . . . 12.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Conclusion A Analysis of PatchMatch (proofs)

A.1 Generic fast patch matching algorithm . . . . . . . . . . . . . . . . . . . . . . . A.2 Convergence of the patch matching algorithms . . . . . . . . . . . . . . . . . . . A.3 Specific PatchMatch algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliography 11

Table 2 :

 2 Comparaison de la performance de débruitage (PSNR moyen) et du temps de calcul moyen par image pour différentes implémentations de BM3D, VBM3D et NL-means sur la base Derf (écart-type du bruit de 20). Il est indiqué si la méthode n'utilise qu'une seule image (IS) ou sur carte graphique (GPU).

			non-local patch search module	1x1 Conv + ReLU	32	1x1 Conv + ReLU	32	1x1 Conv + ReLU	32	1x1 Conv + ReLU	32	DnCNN: 15 layers 64 features 3x3 Conv
	... t-2 t-1	t	t+1 t+2 ...	noisy frame +							residual image
	noisy video sequence	n non-local features:

•

  Global patch search boosts video denoising, Thibaud Ehret, Pablo Arias and Jean-Michel Morel, International Conference on Computer Vision Theory and Applications, 2017 • Automatic Detection of Internal Copy-Move Forgeries in Images, Thibaud Ehret, Image Processing On Line, 2018 • Model-blind video denoising via frame-to-frame training, Thibaud Ehret, Axel Davy, Jean-Michel Morel, Gabriele Facciolo and Pablo Arias, Conference on Computer Vision and Pattern Recognition, IEEE, 2018 • Non-Local Kalman: A recursive video denoising algorithm, Thibaud Ehret, Jean-Michel Morel and Pablo Arias, International Conference on Image Processing, IEEE, 2018 • On the convergence of PatchMatch and its variants, Thibaud Ehret and Pablo Arias, Conference on Computer Vision and Pattern Recognition, IEEE, 2018 • Reducing Anomaly Detection in Images to Detection in Noise, Axel Davy, Thibaud Ehret, Jean-Michel Morel and Mauricio Delbracio, International Conference on Image Processing, IEEE, 2018

• A Non-Local CNN For Video Denoising, Axel Davy, Thibaud Ehret, Jean-Michel Morel, Pablo Arias and Gabriele Facciolo, International Conference on Image Processing, IEEE, 2019 • Video Denoising by Combining Patch Search and CNNs., Axel Davy, Thibaud Ehret, Jean-Michel Morel, Pablo Arias and Gabriele Facciolo, in Journal of Mathematical Imaging and Vision, Springer, 2021 • A Study of Two CNN Demosaicking Algorithms, Thibaud Ehret and Gabriele Facciolo, Image Processing On Line, 2019 • How to Reduce Anomaly Detection in Images to Anomaly Detection in Noise, Thibaud Ehret, Axel Davy, Jean-Michel Morel and Mauricio Delbracio, Image Processing On Line, 2019 • Image Anomalies: A Review and Synthesis of Detection Methods, Thibaud Ehret, Axel Davy, Jean-Michel Morel and Mauricio Delbracio, in Journal of Mathematical Imaging and Vision, Springer, 2019 • Joint Demosaicking and Denoising by Fine-Tuning of Bursts of Raw Images, Thibaud Ehret, Axel Davy, Pablo Arias and Gabriele Facciolo, International Conference on Computer Vision, IEEE, 2019 • GPU acceleration of NL-means, BM3D and VBM3D, Axel Davy and Thibaud Ehret, Journal of Real-Time Image Processing, Springer, 2020

Part I

  according to the value of the distance 7 Keep the N b best elements in L 8 return L

		Search region
	Reference patch	associated to the green candidate
		from frame t
	Second best
	candidate
	in frame t
	Frame t	Frame t+1

  Using optical flow to guide the searchMany video denoising methods take advantage of the optical flow to estimate motion in the video. Typically, optical flow is used by video denoising methods that require aggregating information along motion trajectories [BK95, LF10, BLM16, EMA18, AM19a]. These methods require a motion estimate as accurate as possible. Most patch-based methods, on the other hand, do not require such an accurate motion estimate, as they are based on finding similar patches in a 3D search region[START_REF] Buades | Denoising image sequences does not require motion estimation[END_REF][START_REF] Dabov | Video denoising by sparse 3D transform-domain collaborative filtering[END_REF][START_REF] Maggioni | Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms[END_REF]. These methods either do not use any motion estimate at all, or use a very rough one (e.g. block matching

	1.3 Extensions
	1.3.1

Table 1

 1 elements from L with distance smaller than τ // the 3D transform requires a number of patch which is a power of 2 N l ←closest power of 2 small or equal than min(sizeof L, N ) if L has more than N l elements then

	L ← N l best candidate from L
	return L
	σ Method	Crowd Park Pedestrians station Sunflower Touchdown Tractor	Average
	10 VBM3D (without)	35.52 34.59	40.65	38.38	39.81	39.01	36.82	37.83
	VBM3D (without,d=0) 35.72 35.00	40.13	39.34	40.18	39.01	36.94	38.05
	VBM3D (with)	35.61 34.94	41.04	39.79	41.75	39.89	38.43	38.78
	VBM3D (with,d=0)	35.78 35.19	40.61	40.27	41.85	39.75	38.51	38.85
	20 VBM3D (without)	32.06 31.12	36.81	35.10	35.95	36.05	32.97	34.30
	VBM3D (without,d=0) 32.00 31.24	36.13	35.38	36.09	35.83	33.03	34.24
	VBM3D (with)	32.17 31.44	37.32	36.26	38.02	36.91	34.58	35.24
	VBM3D (with,d=0)	32.10 31.49	36.72	36.32	37.98	36.61	34.58	35.11
	40 VBM3D (without)	28.39 27.64	32.62	31.80	32.31	33.35	29.38	30.78
	VBM3D (without,d=0) 28.29 27.65	32.24	31.79	32.33	33.11	29.40	30.69
	VBM3D (with)	28.55 27.99	33.29	32.80	34.46	34.00	30.79	31.70
	VBM3D (with,d=0)	28.45 27.98	32.94	32.75	34.40	33.79	30.76	31.58

.2: Comparison of the denoising quality with and without guiding with an optical flow. Guiding the search leads to much better results. It is also better in general to keep the additional regularization given by d.

  Comparison of the quality of the matches. The plots show the normalized distance to the ith nearest neighbor i = 1, ..., 300, averaged over 1000 query patches sampled randomly in the bus video.

	0.06	Global exhaustive search
		Local search
	0.05	VP-tree forest search VPLR search
	0.04
	0.03
	0.02
	0.01
	0
	0	50		100	150	200	250	300
	Figure 2.2: Method		σ	n 1	n 2	r 1	r 2
	NLB-local grayscale	all	160	40 + 4 7 (σ -6) max{8, 48 -16 21 (σ -6)} max{8, 16 -4 9 (σ -6)}
	NLB-global grayscale	10	100	60	16	16
		> 10	100	60	16	8
	NLB-local/global color all	350	150	full rank	40 -(σ -10)/3
	BM3D-mix		20 4 × 2 σ/10	32	-	-
		> 20	32	32	-	-
	BM3D-global	all	32	32	-	-
		Table 2.1: Parameters used for the different algorithms.

Table 3

 3 /.7824 37.40/.6681 40.78/.6577 35.62/.8017 39.08/.6103 38.92/.7266 37.52/.7078 VBM4D-np* 33.39/.8237 37.39/.6871 40.56/.7463 35.69/.8457 39.60/.6752 39.93/.7746 37.76/.7588 NL-Kalman 33.34/.8502 36.16/.6782 38.67/.7420 34.29/.8383 38.82/.6940 39.91/.7916 36.86/.7657 .6064 34.60/.5763 36.93/.5579 32.22/.7122 36.09/.4703 35.45/.5689 34.14/.5820 VBM4D-np* 29.55/.6856 34.61/.6073 36.75/.6468 32.07/.7439 36.41/.4795 36.23/.6395 34.27/.6338 NL-Kalman 29.58/.7291 33.19/.5844 35.61/.6444 30.89/.7478 35.91/.5181 36.81/.6868 33.66/.6518 NL-Kalman (oracle) 30.43/.7752 34.18/.6301 36.45/.6738 31.44/.7746 36.99/.6135 37.46/.7116 34.49/.6965 30 VBM3D* 27.59/.4995 32.77/.5224 34.44/.4869 30.14/.6394 34.55/.3906 33.36/.4536 32.14/.4987 VBM4D-np* 27.53/.5988 32.91/.5612 34.45/.5745 29.95/.6704 34.76/.3801 34.14/.5420 32.29/.5545 NL-Kalman 27.30/.6327 31.27/.5335 33.27/.5680 28.64/.6708 33.91/.4034 34.73/.5986 31.52/.5678 NL-Kalman (oracle) 28.48/.6993 32.50/.5802 34.43/.6102 29.44/.7078 35.20/.5186 35.46/.6338 32.59/.6250

	σ Method	Bus	Foreman	Pedestrian area	Crowd_run	Touchdown pass	Station2	Average
	10 VBM3D* 33.32NL-Kalman
	(oracle)	33.87/.8713 36.93/.7230 39.23/.7592 34.64/.8514 39.58/.7433 40.50/.8059 37.46/.7923
	20 VBM3D*	29.57/

.1: Quantitative comparison with VBM3D and VBM4D on grayscale sequences. NL-Kalman (oracle)

Table 4 .

 4 1: Difference between the original NL-means and the improved NL-means version used for this work.

  1 Retrieve the patch stack from positions // Each of the 64 threads starts with a table T of 8 elements containing one row of the stack 2 Apply the 1D DCT on T 3 Apply Algorithm 18 on T 4 Apply the 1D DCT on T 5 Apply Algorithm 19 on T 6 Apply Hadamard on T 7 Apply BM3D's hard thresholding on T 8 Apply Hadamard on T // Hadamard is its own inverse 9 Apply the inverse of Algorithm 19 on T 10 Apply the 1D DCT on T // The normalized DCT is its own inverse 11 Apply Algorithm 18 on T // Algorithm 18 is its own inverse 12 Apply the 1D DCT on T 13 Accumulate the results on num and den with BM3D's weighting scheme

  and 17. Both algorithms are equivalent. Swapping columns and rows (Naive -variant 2) Input: z i , y i thread indices, T thread-specific table of length 8, T table in local shared memory 1 for k from 0 to 7 do 2

	Algorithm 17:
	Algorithm 16: Swapping columns and rows (Naive -variant 1)
		Input: z i , y i thread indices, T thread-specific table of length 8, T table in local shared
		memory
	1 for k from 0 to 7 do
	2	T [64k + 8z i + y i ] ← T [k]
	3 barrier()	// Wait for all threads to finish writing
	4 for k from 0 to 7 do

  table T of size 512 in local shared memory. The table T is organized as a buffer of size [8][8][8] . Each thread (z i , x i ) writes its data so as T [y

  Algorithm 18: Swapping columns and rows Input: z i , y i thread indices, T table of 8 registers for the thread, T table in local shared memory 1 for k from 0 to 7 do

2

T [(64 + x)y

  Each thread starts with the section [z i ][y i ][.] where i is the thread index, and z i , y i thread coordinates. We want the data to be stored as [.][µ i ][ν i ] , with µ i , ν i thread coordinates. We will use local shared memory as intermediate table T of size [8][8/x + 64][8][x] , and assign

reformulate the problem, we start with data as a table T of size [8][8][8] stored in the threads.

  Table 4.3: Compared configurations: The first accelerator is a 16-core CPU, while the other accelerators are GPUs with different levels of performance. The bandwidth and number of operations per seconds were obtained with Clpeak, which estimates the actual peak figures which can be obtained with OpenCL.

	Acc. Description			Bandwidth Float ops
	0	Intel Xeon W-2145 CPU (18.1.0.0920)	63 GB/s	1.8 T/s
	1	INTEL i7-6600U GPU (NEO 18.21.10858) 23 GB/s	380 G/s
	2	AMD Radeon RX 480 (2766.4)		209 GB/s	5.7 T/s
	3	NVIDIA TITAN V (390.129)		611 GB/s	13.7 T/s
		Method	BSD68 Kodak IPOL
		[BCM11]	28.81	29.93 31.89
		OpenCV	28.27	29.33 31.53
		OpenCV GPU 28.24	29.31 31.55
		Ours P5,S1	28.92	30.13 32.60
		Ours	28.51	29.76 32.12

Table 4.4: Comparison of NL-means' denoising performance (average PSNR) on several datasets with the compared implementations (noise standard deviation of level 20).

  Table4.5: Comparison of NL-means' denoising performance (PSNR) and execution time with the compared implementations. CPU methods were run on a single core of an Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz. GPU methods were run on accelerator 3. The running times and the 95% confidence intervals were estimated from 100 runs. Running time for GPU methods corresponds to the sum of time spent running GPU kernels, obtained with nvprof or OpenCL profiling information. Running time for CPU methods corresponds to execution time after the image is loaded and before the result is saved.

	Method	Lena (PSNR-Time) CMLA (PSNR-Time)
	[BCM11]	31.59 -8.5 ±0.2 s		31.75 -522 ±5 s
	OpenCV	31.20 -279 ±2 ms	31.12 -16.0 ±0.3 s
	OpenCV GPU	31.18 -6 ±1 ms		Out of memory
	Ours P5,S1	32.05 -4.27 ±0.02 ms 32.32 -219.2 ±0.5 ms
	Ours	31.86 -0.699 ±0.007 ms	31.99 -20.7 ±0.1 ms
	Accelerator	search	filtering	Total
	0	1672 ±6 ms	282 ±4 ms	1950 ±7 ms
	2	40 ±1 ms	24.0 ±0.3 ms 64.5 ±0.5 ms
	3	15.91 ±0.08 ms 4.44 ±0.02 ms 20.7 ±0.1 ms

Table

4

.6: Comparison of the time taken on the compared accelerators for the main passes (NL-means with our default parameters) on the CMLA. The running times and the 95% confidence intervals were estimated from 100 runs.

  Table 4.6. Most of the running time is spent on the patch search, rather than the NL-means weighting (filtering). The aggregationTable 4.7: Default parameters for σ = 20 for all BM3D methods compared. Parameters in gray can be modified easily by changing a value in the code, or passing an argument when calling the method.

						patch	Number of nearest	Maximum distance for a nearest
	Method	patch size	search region size 2D transform	1D transform	step	neighbors	neighbor
	[DFKE07b] 8 × 8 / 8 × 8 39 × 39 / 39 × 39	Bior/DCT	Haar/Haar	3/3	16/32	2500/400
	[Leb12]	8 × 8 / 8 × 8 33 × 33 / 33 × 33	Bior/DCT	Hadamard/Hadamard	3/3	16/32	2500/400
	OpenCV	4 × 4 / 4 × 4 16 × 16 / 16 × 16	Haar/Haar	Haar/Haar	1/1	8/8	2500/400
	[HK19]	8 × 8 / 8 × 8 23 × 23 / 23 × 23	DCT/DCT	Hadamard/Hadamard	3/3	16/32	3000/400
	[WS17]	8 × 8 / 8 × 8 32 × 32 / 32 × 32	DCT/DCT	DCT/DCT	4/4	8/8	3000/400
	Ours	8 × 8 / 8 × 8 21 × 21 / 21 × 21	Bior/DCT	Hadamard/Hadamard	4/4	8/8	2500/400
			Method	BSD68 Kodak IPOL
			[Leb12]* 29.32	30.68 33.06
			[HK19]*	29.26	30.65 33.05
			Ours*	29.27	30.66 33.06
			[Leb12]	29.50	30.83 33.23
			OpenCV	29.04	30.24 31.82
			[HK19]	29.26	30.61 32.94
			[WS17]	27.34	29.60 31.94
			Ours	29.34	30.65 32.91

Table 4.8: Comparison of denoising performance (average PSNR) of BM3D on several datasets of the compared implementations (noise standard deviation of level 20). *: The default parameters were replaced by the fixed parameters used to compare the three implementations.

  's parameters are almost identical to the original, except for the default search region size.Due to the different sets of 3D filters supported by each implementation, they are not exactly Table4.9: Comparison of denoising performance (PSNR) and execution time between for the compared BM3D implementations. CPU methods were run on a single core of a Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz. GPU methods were run on accelerator 3. The running times and the 95% confidence intervals were estimated from 100 runs. Running time for GPU methods corresponds to the sum of time spent running GPU kernels, obtained with nvprof or OpenCL profiling information. Running time for CPU methods corresponds to execution time after the image is loaded and before the result is saved. *: The default parameters were replaced by the fixed parameters used to compare the three implementations.Table4.10: Comparison of the time taken (in ms) on the compared accelerators for the main passes (BM3D with our default parameters) on CMLA. The running times and the 95% confidence intervals were estimated from 100 runs. *: Using long atomics instead of float atomics

		Method	Lena (PSNR-Time) CMLA (PSNR-Time)
		[Leb12]*	32.98 -7.68 ±0.06 s		33.39 -491 ±5 s
		[HK19]* 32.97 -22.24 ±0.07 ms 33.40 -986.5 ±0.8 ms
		Ours*	32.98 -16.15 ±0.05 ms		33.40 -513 ±4 ms
		[Leb12]	33.04 -6.84 ±0.08 s		33.38 -425 ±3 s
		OpenCV	31.87 -3.36 ±0.07 s		31.71 -204.6 ±0.5 s
		[HK19]	32.94 -17.97 ±0.04 ms 33.07 -760.7 ±0.7 ms
		[WS17]	32.11 -52 ±4 ms		32.67 -5.06 ±0.01 s
		Ours	32.71 -2.59 ±0.02 ms	32.89 -91.9 ±0.5 ms
	Acc.	search	filtering	search	filtering	Total
	0	446 ±5	2106 ±9	5230 ±10	2150 ±10	9930 ±30
	1	1090 ±10	962 ±4	1081 ±5	713 ±4	3860 ±10
	2	17.6 ±0.2	87.2 ±0.2	30.2 ±0.2	105.0 ±0.2 241.9 ±0.2
	2*	17.6 ±0.3	35.3 ±0.4	20.2 ±0.3	38.3 ±0.3 124.6 ±0.5
	3	6.47 ±0.04 33.34 ±0.07 11.75 ±0.05 39.61 ±0.07	91.8 ±0.2

Table 4 .

 4 11 shows the denoising performance and average running time per frame on the Derf

	Method	SF GPU PSNR	Time/frame
	NLM ours	yes	yes 31.54 0.83 ±0.06 ms
	BM3D ours	yes	yes 32.93	4.7 ±0.3 ms
	NLM video ours	no	yes 33.53	10.0 ±0.3 ms
	VBM3D [EA20] no	no 34.21	3.0 ±0.2 s
	VBM3D ours	no	yes 34.31	3.4 ±0.2 ms

Table 4 .

 4 11: Comparison of denoising performance (average PSNR) and average running time per frame of VBM3D and NL-means on the Derf dataset of the compared implementations (noise standard deviation of level 20). Whether the method uses a single frame (SF) or a GPU is indicated.

	Acc.	search	filtering	search filtering	Total
	0	22.5 ±0.7	31.5 ±0.6	48 ±1	72 ±1	174 ±4
	1	28 ±1	20.5 ±0.8	65 ±2	50 ±2	164 ±8
	2	1.7 ±0.1 1.13 ±0.04	3.3 ±0.2 2.6 ±0.3 9.0 ±0.5
	3	0.54 ±0.07 0.55 ±0.05 0.90 ±0.02 1.1 ±0.1 3.4 ±0.2

Table

4

.12: Comparison of the average time per frame taken (in ms) on the compared accelerators for the main passes for our VBM3D implementation (with our default params) on the Derf dataset.

Table 5

 5 Table 5.2: Impact of the architecture of the CNN used for fusing the features.

	# fusion net DnCNN U-Net EDSR
	PSNR	37.51 37.42 37.55
	# stacked frames	1	3	7	11	15
	PSNR	33.88 35.56 36.22 36.42 36.57

.1: Effect of patch size, search region size and number of frames (the number of neighbors is kept equal to the number of frames). All values correspond to the PSNR computed on the validation set for noise with σ = 20. In 5.1a we compare the performance with the one-neighbor-per-frame restriction (ONPF) and without it (free).

Table

5

.3: Impact of the number of frames considered for the video-DnCNN network (the network input is a 3D crop rather than the result of the non-local search). PSNR on the validation set AWGN with σ = 20.

Table 5 .

 5 .1c). 5: Running time per frame on a grayscale 960 × 540 video for VBM3D, DnCNN, VBM4D, VNLB and SPTWO on a single CPU core.

	VBM3D DnCNN VBM4D VNLB SPTWO
	1.3s	13s	52s	140s	210s
	VNLB ViDeNN-G DVDnet FastDVDnet VNLnet
	26.94s	0.186s	2.67s	0.074s	1.16s

  -4 for the rest of the training. Early stopping of the training was done as soon as the training error stagnated for more than three epochs. The model took about one day to train on a NVIDIA Titan X 2 . Table 7.1: Quantitative comparison of different demosaicking methods. The table shows PSNRs on three datasets: the Kodak dataset, the McMaster dataset [ZWBL11] and a subset of the Flickr500 dataset [SCC18]. Table 7.2: Computation time of the compared methods. Computation time was computed for the lighthouse image of the Kodak dataset and is given in seconds.

		Method		Kodak McMaster Flickr500	All
		Gharbi et al. [GCPD16]	41.82	39.08	32.22	38.54
		Tan et al. [TZZZ17]	41.99	39.02	32.08	38.56
		LSSC [MBP + 09]	41.44	36.21	29.86	36.86
		RI [KMTO13]	39.77	34.76	29.46	35.58
		ARI [MKTO15]	39.24	37.48	30.42	36.47
		MLRI [KMTO14]	39.85	35.29	29.62	35.83
		Getreuer [Get12]	39.30	35.95	29.36	35.74
	Method	Gharbi et al. [GCPD16]	Tan et al. [TZZZ17]	LSSC [MBP + 09]	RI [KMTO13]	ARI [MKTO15]	MLRI [KMTO14]	Getreuer [Get12]
	Time	13.97	14.06	736.18	1.86	61.32	1.79	3.07

Table 7

 7

.1 presents the PSNR for all these methods. As one can see the two CNN based methods outperform all previous demosaicking methods. The PSNRs were computed on the Kodak dataset comprised of 24 images, the McMaster dataset from Zhang et al.

[START_REF] Zhang | Color demosaicking by local directional interpolation and nonlocal adaptive thresholding[END_REF]

comprised of 18 images and a subset of 14 images from the Flickr500 dataset from Syu et al.

[START_REF] Syu | Learning deep convolutional networks for demosaicing[END_REF]

  09,[START_REF] Akiyama | Pseudo fourchannel image denoising for noisy cfa raw data[END_REF]. Other methods resort to variational principles

	Original	Gharbi et al.: 36.1dB	Kokkinos et al.: 35.5dB	Ours: 37.3dB

  Table 8.1: PSNR results for different demosaicking methods on the Kodak dataset. Training without ground truth (network from §8.2) outperforms the methods without ground truth while still achieving state-of-the-art PSNRs.

	Method	With ground truth Without ground truth
	Getreuer et al. [Get11]	-	38.1
	Heide et al. [HEK + 14]	-	40.0
	Gharbi et al. [GCPD16]	41.2	-
	Network from §8.2	41.2	41.3

  Algorithm 20: Anomaly detection algorithm input : Input image u, reference image v, size of patch sp, number of nearest neighbors n, h the patch similarity parameter, a list R of disk radii to test, the NFA threshold ε, a neural network nn (optional) output : List of detections l (contains size and position)

	1 (u i ) ← multiscaleDecomposition(u, nscales) // See Section 9.3.2
	2 (v i ) ← multiscaleDecomposition(v, nscales) // and Algorithm 26
	3 for scale i = 0 to 3 do
	4	if nn is provided then
	5	ûi ← extractF eatures(u i , nn) // Use neural network features
		to represent u i
	6	vv ← extractF eatures(v v , nn) // See Section 9.3.3
	7	else
	8	ûi ← u i
	9	vi ← v i
	10	r i ← computeResidual(û i , vi , sp, n) // See Algo. 24
	11	r i ← f itResidual(r i ) // See Algo. 7
	12	l i ← detect(r i , height(û 0 ), width(û 0 ), channels(û 0 ), R, ε) // See algo.
		27
	13

  Detecting anomalies in a residual image input : A residual image r, (u w , u h , u c ) the size of original image u (before any downsampling), the list R of disk radii to test, the NFA threshold ε, sp the patch size used for Alg. 24 output : A list of detections l 1 for R i in R do

	image. This leads to
			N =	4u h u w 3	N chan N conv	(9.10)
	The detection process is summarized in Algorithm 27.
	Algorithm 27: detect: 2 Compute the disk kernel function f of radius R i
	3	r ← r * f i // Convolve the image with the kernel (exact
		convolution done in Fourier)
	4	Estimate σ the standard deviation of	r
	5	for each pixel rk of r outside of the border of size sp/2 // Detections are
		unreliable on the border
	6	do
	7	if log 4u h uwuc 6	length(R) erfc |r k | σ	< ε // See Eqs (9.4) and (9.10)
	8	then
	9
		N scales -1 i=0	|Ω s | = 4u h uw 3	with u w and uh the width and the height the input

Le démosaïquage, étudié dans le Chapitre 7, et le débruitage sont les deux premières étapes de la chaîne de traitement d'image de n'importe quelle caméra. Elles sont la clé pour avoir des images couleurs de bonne qualité. Un nouvel axe prometteur de recherche propose de résoudre les deux problèmes à la fois en utilisant des réseaux de neurones convolutionnels. Étant donné qu'il est très difficile, voire impossible, d'obtenir une vérité terrain pour des images naturels, ces réseaux ne sont pas entraînes avec des données brutes réelles. À la place, des données simulées sont utilisées.Dans ce chapitre nous présentons une méthode qui permet d'apprendre un réseau directement à partir d'images brutes encore mosaïquées, sans avoir besoin de vérité terrain couleur. Nous appliquons cette méthode pour apprendre un démosaïquage et débruitage conjoint directement depuis des données brutes. Nous montrons aussi que cela peut être appliqué pour raffiner un réseau sur une acquisition rafale spécifique pour améliorer la qualité globale de la restauration. Un exemple sur des données synthétiques est présenté dans la Figure19.

Available at http://www.cs.tut.fi/~foi/GCF-BM3D/.

https://media.xiph.org/video/derf/.

This corresponds to Matlab's imresize scaling function using the lanczos3 interpolation kernel.

https://media.xiph.org/video/derf/

The color sequences are from https://media.xiph.org/video/derf/. The grayscale sequences are from http://www.cs.tut.fi/~foi/GCF-BM3D/, except for football and mobile, which have been obtained by averaging the channels from the corresponding RGB sequences.

For V-BM3D we show only grayscale results since there are no Linux binaries for the color version of V-BM3D.

http://www.cs.tut.fi/~foi/GCF-BM3D/

T [k] ← T [64y i + 8z i + k]

Available on http://mcolom.info/download/no_noise_images/no_noise_images.zip

https://media.xiph.org/video/derf

The code to reproduce our results, the training and testing datasets can be found at https://github.com/ axeldavy/vnlnet.

We used the implementation of https://github.com/lessw2020/ Ranger-Deep-Learning-Optimizer

The median and mode are taken element-wise. For a continuous random variable the L0-loss is defined as a limit. See[START_REF] Kay | Fundamentals of statistical processing, volume i: Estimation theory: Estimation theory[END_REF] and [LMH+ 18].

https://media.xiph.org/video/derf/

https://www.army.mil/

This description of the network training is based on the authors' description. We did not attempt to emulate this training. The neural network weights used here therefore are the original ones provided by the authors.

This description of the network training is based on the authors' description. We did not attempt to emulate this training. The neural network weights used here therefore are the original ones provided by the authors.

http://r0k.us/graphics/kodak/

û(j) = i∈{s|j∈Ws} Pi (j) #{s|j∈Ws}// Aggregate the different estimates

return r(u) = û -u

Notation convention: W s is the set of pixels in the patch centered at s. Pi (j) is the value at pixel j of the reconstructed patch centered at i.

The vertices in the propagation graph are indexed using a topological ordering (such an ordering exists because the graph is acyclic). The propagation follows this ordering. The complete definition of the propagation graph is the following, and it fully specifies the propagation in a PatchMatch algorithm.

This scheme differs from the one considered in Proposition 10.1, which takes k equally distributed samples Sη ∼ Q (η, •). But the theory applies the same as long as C is in accordance with the sampling procedure.

The kernel needs to be normalized by the area of the portion of the kernel that fits in the image domain.

Remove any detection at the boundary (of size sp) of EThe next steps of the detection correspond to multiple post-processings trying to minimize false alarms and to adjust the detection mask to the actual forged regions.

Remerciements

Je remercie aussi mes rapporteurs (Giacomo Boracchi et Karen Egiazarian

Gharbi et al.

+ our fine-tuning

Figure 8.12: Experiment on a real burst. Left to right: The result of the HDR+ pipeline [HSG + 16], bilinear interpolation, Gharbi et al.

[GCPD16], and Gharbi et al.

[GCPD16]

with our fine-tuning. Contrast in dark regions was enhanced for all methods. Note that the HDR+ pipeline includes color balance as well as sharpening. It also uses all the images of the burst to produce the result (all other methods use only the reference frame). Figure best visualized zoomed in on a computer.

HDR+ Bilinear

Gharbi et al.

Gharbi et al. + our fine-tuning

Figure 8.13: Experiment on a real burst. Left to right: The result of the HDR+ pipeline [HSG + 16], bilinear interpolation, Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF], and Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] with our fine-tuning. Contrast was enhanced for all methods except HDR+. Note that the HDR+ pipeline includes color balance as well as sharpening. It also uses all the images of the burst to produce the result (all other methods use only the reference frame). [GCPD16], and Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] with our fine-tuning. Contrast in dark regions was enhanced for all methods. Note that the HDR+ pipeline includes color balance as well as sharpening. It also uses all the images of the burst to produce the result (all other methods use only the reference frame). over-represented. To equalize the residuals and simplify the detection model, we use a non-linear transform and rescale r(u) in order to fit a centered Gaussian distribution with unit variance.

Statistical detection by the a contrario approach

The a contrario model. Our goal is to detect structure in the feature image residual r(u) = û -u. We are in a much better situation modeling r(u) than u. Indeed, contrarily to u, r(u) is by construction unstructured and akin to a colored noise. We treat it as a stationary probabilistic spatial process and follow [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF], who proposed automatic detection thresholds in any colored Gaussian noise. Their a contrario framework computes a Number of False Alarms (NFA) for each value in excess in a colored noise image, under the null hypothesis that the residual is centered Gaussian noise of unit variance. The null hypothesis doesn't require the noise to be uncorrelated. Since anomalies are expected to deviate from this background model, this amounts to checking the tails of the Gaussian and to retain high values as significant if their tail has a very small area. Computing an NFA is justified by the intensive multiple testing involved in the detection. Indeed every pixel of each residual channel are tested.

More precisely, given a set of random variables 24.2 1.0 70 100 [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] 65.0 1.0 100 100 Table 9.2: This table is similar to Table 9.1, but in this case each method detection threshold is set so as there are 1% false positives. Hence, the criterion is to detect as many anomalies as possible (third column) while having a high true positive rate. The winners are clearly [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] and [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF]. 9.1. Table 9.2 also shows that having a 1% detection is useful to obtain a good precision but leads to almost all images getting false positives. In practice 1% is too large a tolerance for images. In Figure 9.6 we show the result of the detections on 9.5 corresponding to Table 9.1 for the different methods.

Computation time analysis

In this section we do a brief computation time analysis. All algorithms have wildly different computation times. For example [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF] method is really fast as no really complex computations are needed. On the contrary the [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] method is really slow. Table 9.3 summarizes the computation time for the different algorithms for the parameter used for the experiment. It's worth noting that for the larger parameters the [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] method requires many hours to compute a single result. It's also worth noting that even though the [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] and [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] algorithms are not the fastest ones, the dictionaries of patches and indexes for the searches can be precomputed and therefore accelerated for fast industrial applications. For example the processing of [START_REF] Boracchi | Novelty detection in images by sparse representations[END_REF] only takes 12s when the dictionary is prelearned. The computation time estimation was done on a core i7-7820HQ 2.90GHz using authors' code whenever it was available ( [BCW14], [START_REF] Davy | Reducing anomaly detection in images to detection in noise[END_REF] and [START_REF] Mishne | Multiscale anomaly detection using diffusion maps and saliency score[END_REF] are multithreaded so actual computation times are reported. We report 1/8 of the actual computation time for [START_REF] Aiger | The phase only transform for unsupervised surface defect detection[END_REF], [START_REF] Grosjean | A-contrario detectability of spots in textured backgrounds[END_REF] and [START_REF] Zontak | Defect detection in patterned wafers using anisotropic kernels[END_REF] for a fair comparison). . When available, red is the detection with the threshold corresponding to the lowest NFA. For [MC14] we adopted a similar color coding: white between 0 and 0.5, cyan between 0.5 and 0.7, green between 0.7 and 0.9 and orange above 0.9. In this chapter we study the convergence of PatchMatch and its variants, and derive bounds on their convergence rate. We consider a generic PatchMatch algorithm from which most specific instances found in the literature can be derived as particular cases. We also derive more specific bounds for two of these particular cases: the original PatchMatch and Coherency Sensitive Hashing. The proposed bounds are validated by contrasting them to the convergence observed in practice. This work was published in [START_REF] Ehret | On the convergence of patchmatch and its variants[END_REF].

Introduction

Patch-based methods are among the state-of-the-art in several image/video processing and computer vision applications. Often these methods require finding for all patches of a query image, the (approximate) k nearest neighbors among the set of patches of a database image. This is referred to as an approximate k nearest neighbors field (k-ANNF) from the query image to the database image.

Examples of the application of k-ANNFs can be found for image completion (and editing) [START_REF] Arias | A variational framework for exemplar-based image inpainting[END_REF][START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF], denoising of images [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF] and video [START_REF] Ben-Zrihem | RIANN: Approximate Nearest Neighbor Fields in Video[END_REF][START_REF] Liu | A high-quality video denoising algorithm based on reliable motion estimation[END_REF], video stylization [BZL + 15, BZZM15], alpha matting [HRR + 11], optical flow [BYJ14, FBK15, HBK + 14, LYMD13] and stereo-vision [START_REF] Lu | Patch match filter: Efficient edge-aware filtering meets randomized search for fast correspondence field estimation[END_REF][START_REF] Bleyer | Patchmatch stereostereo matching with slanted support windows[END_REF]. Also in the close field of computer graphics ANNFs of 3D surface patches have been applied to mesh tracking [START_REF] Klaudiny | Cooperative patch-based 3D surface tracking[END_REF] and texture transfer [START_REF] Chen | Nonparametric texture transfer using meshmatch[END_REF].

The brute-force computation of the k-NNF scales linearly with the product of the number of pixels in the query and the database images, and is therefore prohibitively slow. The first practical approaches rely on data structures such as hash tables or partition trees (see [START_REF] Kumar | What is a good nearest neighbors algorithm for finding similar patches in images?[END_REF] and references therein). The approximate k nearest neighbors of each query patch are computed independently. Even if these approaches greatly improve with respect to the brute-force search, they are still too slow (and moreover scale badly with the patch size) for many applications such as those requiring user interaction.

The introduction of the PatchMatch algorithm [START_REF] Barnes | Patchmatch: a randomized correspondence algorithm for structural image editing[END_REF] and its generalized version

Size Filter

The first post-processing step tries to remove "unlucky" matches. Sometimes good matches can be created purely by chance, but in practice these fortuitous regions are very small. The application of a filter which cuts connected components whose area is below a certain threshold removes these "unluckily" matched regions. Such a filter is presented in Algorithm 33.

Algorithm 33: sizeFilter algorithm input : A detection mask M , area threshold τ s output : A filtered detection mask M

Minimum Displacement Filter

As a last step to remove false alarms, candidates that are too close in the image plane are filtered out. If the copy is too close to the original, humans detect easily the forgery. Thus, if a detected forged region is too close to its copy, both are considered false alarms and therefore discarded. This idea is synthesized in Algorithm 34. While this post-processing is necessary for the method to work well, it's actually redundant with the exclusion region from PatchMatch. Indeed setting τ m to the maximum between τ m and τ d forces all matches of PatchMatch to be at a distance superior or equal to τ d . Therefore there would be no matches to be rejected with this displacement filter. The deliberate choice to still present the displacement filter was made to describe the originally proposed post-processing steps, without mixing ideas from different sections.

Symmetrization of Detections

Because the process is not entirely symmetric, it can happen that both regions detected as forged have different shape and size. It may even happen that only one of the forged regions is detected while the other one is missing from the detection mask. To avoid this problem, the detection mask is symmetrized using the displacement map. This process is presented in Algorithm 35. This section is necessary even though PatchMatch already has a mechanism favoring the symmetry. Indeed PatchMatch only incites it in the sense that if the symmetric match isn't a better candidate then it's rejected. Therefore at the end of the PatchMatch step, the displacement can be asymmetric (even though in practice it is already quite symmetric). Because a forgery is necessarily symmetric, the final detection map must be symmetric. Therefore the detection regions in the detection map are symmetrized using the method presented in this section.

Dilation

Finally, for the last step, each detection is slightly dilated. Indeed both the median filter and the error filter tend to reduce the size of a detected region. This dilation step mitigates these effects. This step is presented in Algorithm 36.

Algorithm 36: dilationFilter algorithm input : A mask of detections M , the radius of the filter ρ output :

Experiments

In this section, results showing both the strengths and the limits of the method are presented. The displacement results presented in this section will have the same visualization. The color scheme used to represent the direction and magnitude of the displacement is shown in Figure 11.4. a contrario stochastic model on the perturbation. We will consider for now that D i and D j are derived from the same patch but one of them has been corrupted by Gaussian noise of variance σ 2 i.e. since the descriptors consist of gradients ∀k, l, D j k,l = D i k,l + n k,l where n k,l ∼ N (0, 2σ 2 ) and are independent. Matching both descriptors requires that max k,l n 2 k,l τ . The probability of matching in this case is then

where n follows a χ 2 distribution with 1 degree of freedom. We can therefore control the number of false detections by choosing the proper τ according to

where is the number of false alarms per N tests number of tests and chi2inv is inverse of the χ 2 cumulative distribution function. The main point of formula (12.3) is that it reduces the initial method dependency on many detection parameters to just one, namely σ. We can argue that this last one is not critical. Indeed, even though the dependency on σ is strong, as long as the degradation is not too large, there will be a scale in which σ is small enough so the detection will work: indeed σ is divided by two at each octave in the SIFT method. For example, this exigent threshold can work for a noise of 4, but requires the tampered area to be four times larger for a detection. It might be objected that zooming down also makes naturally similar objects become more similar. Yet our experiments indicate that this is not the case, indeed their small but significant differences encompass all scales. To summarize, granting that we allow for one false detection on average on a set of images, the method is parameterless as it adapts to the number of tests and to the patch size. Of course it might be coupled with an automatic noise estimator to give an good guess of σ. Assuming a perturbation noise of variance σ 2 = 1 and the use of descriptors of size 4 × 4 × 3 (derived from 6 × 6 color patches), a number of false alarms of = 1 and testing on 100 images with on average 50 keypoints (this corresponds to the COVERAGE dataset presented in Section 12.3), Equation (12.3) gives τ = 2.9. The advantage of using color descriptors in this case is either to increase the size of the descriptor (allowing for a larger τ and detecting more) or to reduce the spatial size for a same size of descriptor (allowing to detect smaller forgeries).

An interesting side effect is that this test is really fast to compute. Indeed to detect forgeries each keypoint must to compared against all others. Since we are comparing keypoints inside a single image this gives (K-1)(K-2) 2 pairs to be tested, where K is the number of descriptors. (Of course all descriptor self-matches are discarded). For large images the computation of the distance becomes quickly a bottleneck for distances that are costly. In our case it is not necessary to compute d max before doing the test, the test can be done during the computation of d max which allows for early stopping. Since most keypoints won't match, the number of operations done per comparison is in practice much smaller than the size N 2 of the descriptor. An experimental verification of this fact is made in Section 12.3.

Finally we need to take into account all possible flips for the forged regions. While the matching process doesn't detect flips it is possible to still detect them at the cost of a few more computations. The modified distance to test flips is then

where D k,l = (x k,l , y k,l ). Indeed when flipped, the indexes in one direction are reversed but also the gradients in that direction are opposite. into account by just testing the flip in one direction (in our case in the y direction). In the end, we test each pair of keypoints with both distances to take into account flips. Having to do twice the computation is not a problem in practice as each test is very efficient.

Experiments

In this section the images are all shown in grayscale even though they are originally in color. This allows for a better visualization of the matches. Nevertheless, the descriptors were color descriptors. We present results on three different datasets: GRIP [START_REF] Cozzolino | Efficient dense-field copy-move forgery detection[END_REF], Image Manipulation (IM) [CRJ + 12] and COVERAGE [WZS + 16] which is the dataset that inspired this study as it focuses on distinguishing forgeries from similar but genuine objects. All images shown in this section come from these datasets. We decided to use descriptors of size 3 × 8 × 8 for the IM dataset and 3 × 4 × 4 otherwise as the images from the IM dataset are much larger than the ones from the other datasets. Indeed the size of the descriptors needs be chosen so to be smaller than the expected size of the forged regions. Each time the threshold was computed using Equation (12.3) from Section 12.2.3. We also verified that the number of comparisons done to compare two descriptors was much smaller than the size of a descriptor. For example for the image shown in Figure 12.1, of size 424 × 421 and containing 207 descriptors only 1.1 comparisons were necessary on average for a descriptor size of 3 × 6 × 6 = 108 that is almost the size of the descriptor used for SIFT. Thus the detection is really fast even for large images with a large number of keypoints.

Table 12.1 shows that while the methods focuses on being robust to similar objects and reduces as much as possible false detections, it is actually competitive with previous keypoint based methods. Moreover, the number of false alarm is definitely under control : only very little false alarms were found in all three datasets. One of these false detection is shown in Figure 12.3. We also verified that while robust to similar objects (and therefore very precise) the method still is robust to reasonable noise and compression.

Figure 12.2 shows different examples of successful detections. The method is able to detect well rotation, uniform illumination changes, scaling and compression. As can be seen, the more

A Analysis of PatchMatch (proofs)

A.1 Generic fast patch matching algorithm

A.1.1 Random search

In the following, we will say that η ∈ {U z α}, η being a k-set of elements, if and only if U z (η) α (U z being defined in Def. (10.1)).

Lemma 11.1. For all z ∈ A, C(z, .) is a non-increasing function such that for ε > 0, C(z, ε) ∈ [0, 1[ and C(z, ε) = 1 for ε 0.

Proof. Let us recall the definition of C:

The function C(z, •) is a supremum of probabilities (in fact it is a maximum because the set {U z ε} is finite). We remind the property for a candidate η, Q(η, L) < 1 if the set of acceptable list of candidates L is not empty. Thus C(z, ε) ∈ [0, 1[, for all z ∈ A and ε > 0. Since inf U z = 0, the probability of transitioning to a negative energy is 0, and thus

(A.2) and therefore C(z, ε 1 ) C(z, ε 2 ).

A.2 Convergence of the patch matching algorithms

For the proof of Lemma 10.2, we will use the following lemma and its subsequent corollary.

Lemma A.1. Let y, x be real random variables (x with pdf f according to Lebesgue measure µ) and E an event of the form s i ∈]a i , +∞[, a i ∈ R, for i ∈ 1, N . We assume that y is conditionally independent from E given x. Then for any Y, X:

Proof. The proof follows from Bayes rule and the conditional independence between y and E:

(A.4)

Corollary A.1. Let y 1 , . . . , y n , x 1 , . . . , x n be real random variables and E an arbitrary random event of the form defined in Lemma A.1. We assume that for all i ∈ 1, n , given x i , y i is conditionally independent from E, x j and y j , for j = i. Then for any Y i , X i :

Proof. Using Bayes rule,

where the last inequality is given by the application of Lemma A.1 on each term of the product.

A.2.1 Energy decay in the nth step

Lemma 11.2 (Constraints propagation). Consider an assignment ϕ n+1 resulting from an iteration of Algorithm 28. Then for each pair of nodes x, z ∈ V,

where the levels ε z,x 0 are as follows. For the ancestors of x (i.e. P(z, x) = ∅) the levels ε z,x are defined via the following recursion starting from x and following the inverse propagation order:

For the rest of the nodes ε z,x = -1.

Proof. We start by observing the following property of the merge operator. Consider a node y ∈ V and two sets of candidates patches ξ, η. Then it is easy to show that

The proof of the lemma is trivial for the nodes that are not ancestors of x. To prove it for the ancestors we proceed by induction starting from x and following the inverse propagation order. Let z, a node in the query image A, be an ancestor of x. We assume that the statement holds for the nodes preceding z in the inverse propagation order (or equivalently those succeeding z with the propagation order).

In particular let y be a child of z, i.e. z ∼ y, and assume that ϕ n+1/2 y ∈ {U y ε y,x }. The candidates ϕ n+1/2 y result from the propagation from the parents of y, among them is z. By (A.11) it follows that if U y (A z,y ϕ n+1 z ) < ε y,x , then U y (ϕ n+1/2 y ) < y,x , violating our assumption. Thus, necessarily

Since this holds for all children y of z, we have that

In the case of the nearest neighbor search, we can directly use the L ε y,x s instead of having to use the upper level sets defined by the ε y,x s in the following proof for a tighter bound.

Theorem 11.1 (Point-wise convergence). Consider the field of candidate matches at iteration n, ϕ n . Define ϕ n+1 by applying an iteration of the Generic PatchMatch in Algorithm 28. Then, for all ε > 0, for all x ∈ A, we have

where µ(z) was defined in (10.5) as the number of parents of node z and C i denotes the worst case transition probability for kernel Q i , as in Eq. (10.8).

Proof. We consider a topological ordering of the nodes in the query image. Given two nodes z, y ∈ A, we use the notation z < y if z precedes y. We denote by y -1 and y + 1 the nodes before and after y in the ordering.

The proof consist on a recursion on the ordered set of nodes. For y ∈ A we define the following events:

The event S y restricts the candidates at iteration n of the nodes succeeding y, whereas the event P y considers the candidates at iteration n + 1 of the nodes preceding y. From Lemma 11.2 we have that:

The last equality holds because for z > x as the level ε z,x is defined as -1. Therefore the conditions over ϕ n+1 z or ϕ n z for such nodes are trivially satisfied. We proceed by showing that the following recursive relation holds for any node y: P(P y ; S y ) P(P y-1 ; S y-1 )C 1 (y, ε y,x )C 2 (y, ε y,x ) µ(y) .

(A.18)

The result then follows by applying this recursion backwards from x until the first node in the topological ordering. To show (A. C 2 (y, {U y ε y,x }) µ(x) P(P y-1 ; S y-1 ; ∀z ∼ y , U y (Aϕ n+1 z ) ε y,x ) C 2 (y, {U y ε y,x }) µ(x) P(P y-1 ; S y-1 ).

(A.22)

In the third step we have applied Corollary A.1, whereas the forth step results from the definition of C 2 . The recursion (A.18) follows from (A.20) and (A.22).

Remark A.1. The proof is based on the fact that, due to the propagation, restricting the energy of the candidates at x implies constraints on the candidates of its ancestors (Lemma 11.2). We then bound the probability of all random samples drawn to satisfy those constraints. The more restrictive those constraints are, the smaller the probability of satisfying them. is a set of k-sets, and it is difficult to evaluate the probability of sampling a k-set in that set. This difficulty disappears for k = 1. Then L ε z,x ⊆ B, and computing the probability of sampling in the allowed set becomes easier. Thus a tighter bound can be computed for k = 1.

Proof. The bound in Theorem 11.1 applies. We now express C 1 in terms of the new C . For that, we compute an upper bound for P(U z (S 1 ϕ n z ) a | U z (ϕ n z ) a, E) with a 0 and E any event, ∀z ∈ A. We first remark that, if S i 1 ϕ is the i th sample generated from the random sampling around ϕ,

Let us remind that η and Sη are k-sets, i.e. sets of k distinct candidates. The candidates in Sη are sampled centered at the best candidate in η (the one minimizing U z ). We know that η ∈ {U z a}, which only constrains the worst candidate in η, but says nothing about the best candidate. This is why for this proof there is no restriction on where the sample comes from. Therefore, with

Since the support of the random search is the full image, it guarantees that ∀z ∈ A, ∀a > 0, C (z, a) < 1. This implies that the bound found is strictly inferior to one and therefore the convergence is insured.

Corollary 11.2. In the case of the search of the nearest neighbor, the upper bound can be written as

This bound is actually tighter than the one derived in [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF].

Proof. The derivation of the tighter bound presented in Equation (A.38) is the same as in Proposition 11.1. The difference comes from the that in the case k = 1, the best current match is also the worst current match therefore the element used to sample is necessarily with an energy larger than ε z,x .

The second part of the proof concerns the comparison with the bound of [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF]. The difference between the bound (A.38) and the one in [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF] lies in the levels ε z,x inside the factors C . The levels used in [START_REF] Arias | Analysis of a variational framework for exemplar-based image inpainting[END_REF] are of the form εz,x , where z,x is defined as follows:

with P(z, x) being the set of all the paths from z to x in the graph and

A ∞ Therefore, due to the monotonicity of C we just have to show that ε z,x εz,x .

For z = x, ε z,x = ε and z,x = 0 therefore the property is verified. Suppose now that the property is true for any y such that y > z. We will show that in this case the property stays true for z. Suppose that ε

such that U z (η) < εz,x . Let y be the child of z such that z,x = y,x + d z,y . In this case we have that

We then have found η ∈ L ε y,x i.e. Aη ∈ {U y ε y,x } therefore ε y,x U y (Aη) < εy,x which is contradictory with the hypothesis. The property is then also valid for z.

Remark A.2. The tighter bound for the specific case of k = 1 mentioned in the Remark after the proof of Theorem A.15 applies in this case as well, yielding a bound for the original PatchMatch that is tighter than the one from Proposition 11.1. We compared both bounds and in practice the difference between them is small. For our experiments in the paper we have used a bound similar to the one of the Proposition 11.1.

A.3.2 The CSH algorithm

Because the h i s are independent, P H (h 1 (p) = h 1 (q), . . . , h n (p) = h n (q)) = n i=1 P H (h i (p) = h i (q)) (A.44)

Using the (R, cR, p 1 , p 2 )-sensitive property of H, for all i P H (h i (p) = h i (q)) = 1 -P H (h i (p) = h i (q)) (A.45) 1 -p 1 (A.46) Therefore P G (g(p) = g(q)) (1 -p 1 ) n (A.47) and P G (g(p) = g(q)) 1 -(1 -p 1 ) n . (A.48)

Using similar derivation, the corresponding result can be proved for the second part of the sensitive definition. This result in G an OR family function being (R, cR, 1 -(1 -p 1 ) n , 1 -(1 -p 2 ) n )sensitive.

Proposition 11.2. For a (R, cR, p 1 , p 2 )-sensitive family of hashing functions such that R max z∈A K z (see Definition 10.1), the sequence (ϕ n ) defined by the CSH algorithm converges in probability to a minimizer of the total energy, in the sense that lim n→∞ P(U x (ϕ n x ) ε) = 0, ∀ε > 0, x ∈ A, (A.49) with a geometric convergence rate. Moreover for all ε > 0, for all x ∈ A, P(U x (ϕ n+1 x ) ε) P(U x (ϕ n x ) ε) z∈A C 2 (z, {U z ε z,x }) µ(z) f (p 1 , ε z,x }), (A.50)

where f (α, β) = (1 -α k ) if ε z,x > 0, 1 otherwise.

From a set of LSH hash functions H, a OR family of function G can also be defined. The function g ∈ G is based on a set of n random functions h 1 , . . . , h n from H such that for all p, q, g(p) = g(q) if and only if there exist i ∈ 1, n such that h i (p) = h i (q). Proof. Let R max z∈A K z and H an (R, cR, p 1 , p 2 )-sensitive family of functions. Consider also G an OR family function based on H so that a function from G is generated using at least k functions from H. An upper bound for P(U x (S 1 ϕ n x ) a | E) with a > 0, ∀x ∈ A and E an undefined event will now be derived. R is then chosen such that for all x ∈ A, x -N k (x) R, where N k (x) ∈ B is the k th nearest neighbor of x. This property is true for the l th nearest neighbor N l (x), with l k. Therefore we have P(h(N l (x)) = h(x)) p 1 . Moreover the binning is independent from the current matching ϕ n

x .

For a given h ∈ H and H ⊂ H such that h ∈ H , for a query q, for all p ∈ B h(q) = h(p) ⇒ ∃h ∈ H , h (q) = h (p) (A.52) Therefore for a list H = {h 1 , . . . , h k } ⊂ H, for a query q, for all p ∈ B h 1 (q) = h 1 (p 1 ), . . . , h k (q) = h k (p n ) ⇒ (∃h ∈ H , h(q) = h(p 1 )), . . . , (∃h ∈ H , h(q) = h(p k )) (A.53) and P(h 1 (q) = h 1 (p 1 ), . . . , h k (q) = h k (p n )) P((∃h ∈ H , h(q) = h(p 1 )), . . . , (∃h ∈ H , h(q) = h(p k ))) (A.54) When working with the elements of the OR family G, we can define H as the set functions from H used to generate g (H verifies the properties of the previously used H because at least k functions are used to generate each element of G), P G (g(q) = g(p 1 ), . . . , g(q) = g(p n ))

= P H ((∃h ∈ H , h(q) = h(p 1 )), . . . , (∃h ∈ H , h(q) = h(p n )))

P H (h 1 (q) = h 1 (p 1 ), . . . , h n (q) = h n (p n )) (A.55) which leads to P G (g(q) = g(p 1 ), . . . , g(q) = g(p k )) P H (h 1 (q) = h 1 (p 1 ), . . . , h k (q) = h k (p k )) (A.56) because the h i are independent P G (g(q) = g(p 1 ), . . . , g(q) = g(p n ))

P H (h i (q) = h i (p i )) p k 1 (A.57) 244 when using the original LSH property for H. This implies that P(U x (S 1 ϕ n x ) = 0) = P G (g(q) = g(p 1 ), . . . , g(q) = g(p n )) p k 1 (A.58)

Finally, P(U x (S 1 ϕ n x ) a | E) 1-P(U x (S 1 ϕ n x ) a | E) 1-P(U x (S 1 ϕ n x ) = 0 | E) 1-p k 1

Replacing C 1 (z, ε), for z ∈ A and ε > 0, by (1 -p k 1 ) in the proof of Theorem 10.1 gives the result in Proposition 10.2.