Yu Guo

Qiyu Jin

Jean-Michel Morel

Gabriele Facciolo

How to best combine demosaicing and denoising?

Keywords: 2020 Mathematics Subject Classification. Primary: 68U10; Secondary: 62H35 Demosaicing, denoising, pipeline, image restoration

Image demosaicing and denoising play a critical role in the raw imaging pipeline. These processes have often been treated as independent, without considering their interactions. Indeed, most classic denoising methods handle noisy RGB images, not raw images. Conversely, most demosaicing methods address the demosaicing of noise free images. The real problem is to jointly denoise and demosaic noisy raw images. But the question of how to proceed is still not yet clarified. In this paper, we carry-out extensive experiments and a mathematical analysis to tackle this problem by low complexity algorithms. Indeed, both problems have been only addressed jointly by end-toend heavy weight convolutional neural networks (CNNs), which are currently incompatible with low power portable imaging devices and remain by nature domain (or device) dependent. Our study leads us to conclude that, with moderate noise, demosaicing should be applied first, followed by denoising. This requires a simple adaptation of classic denoising algorithms to demosaiced noise, which we justify and specify. Although our main conclusion is "demosaic first, then denoise", we also discover that for high noise, there is a moderate PSNR gain by a more complex strategy: partial CFA denoising followed by demosaicing, and by a second denoising on the RGB image. These surprising results are obtained by a black-box optimization of the pipeline, which could be applied to any other pipeline. We validate our results on simulated and real noisy CFA images obtained from several benchmarks.

1. Introduction. Most portable digital imaging devices acquire images as mosaics, with a color filter array (CFA), sampling only one color value for each pixel. The most popular CFA is the Bayer color array [START_REF] Bayer | Color imaging array[END_REF] where two out of four pixels measure the green (G) value, one measures the red (R) and one the blue (B). The two missing color values at each pixel need to be estimated for reconstructing a complete image from a CFA image. The process is commonly referred to as CFA interpolation or demosaicing. CFA images have noise, especially in low light conditions, so denoising is also a key step in the imaging pipeline.

Denoising and demosaicing are often handled as two independent operations [START_REF] Paliy | Demosaicing of noisy data: spatially adaptive approach[END_REF] for processing noisy raw sensor data. Most of the literature addresses one or the other operation without discussing its combination with the other one.

All classic demosaicing methods have been proposed for noise free CFA images, while denoising algorithms have been designed for color or gray level images only considering additive white noise. Yet the input data is in reality different: it is either a CFA image with noise, or a demosaiced image with structured noise. Therefore, we can distinguish three main pipeline strategies: denoising first followed by demosaicing (DN &DM ), demosaicing first followed by denoising (DM &DN ), and joint demosaicing-denoising. It might be argued that with the advent of deep learning, the joint operation will become standard and the first two solutions obsolete. But there are three good reasons to address them. The first one is that, contrary to classic image processing chains, processing chains based on deep learning remain domain and device dependent. In other terms, even if they can give the best results on a given test set or device, there is not guarantee that they will deliver good results on out of domain images, or on new devices. Hence, even with slightly apparent lower performance, classic algorithms still retain their value. Secondly, as has been verified many times, insight obtained by combining classic algorithms leads to conceive better deep learning structures. Last but not least, classical algorithms are characterized by computational efficiency and suitability for acceleration. This is exemplified by the successful implementation of classical algorithms, such as the BM3D algorithm, on select mobile devices, made possible through the adoption of advanced process chips, along with continued efforts in algorithmic enhancement and optimization. This accomplishment underscores the promising potential for classical algorithms to extend their reach to a broader spectrum of edge computing devices in the foreseeable future. In contrast, the computational demands of neural networks present challenges when it comes to deployment on low-performance hardware. For these reasons, we shall focus here on a comparison of denoising first followed by demosaicing (DN &DM ) with demosaicing first followed by denoising (DM &DN ), and to generalizations of both approaches.

Currently, the most popular classic pipeline is the DN &DM scheme. This is determined by two basic assumptions. First, after demosaicing, the noise becomes correlated and no longer retains its independent identically distributed (i.i.d) white Gaussian properties. This has a negative impact on traditional denoising algorithms that rely on additive white Gaussian noise (AWGN). Second, state-of-the-art demosaicing algorithms are often designed on a noise-free basis. As a result, many state-of-the-art works [START_REF] Paliy | Demosaicing of noisy data: spatially adaptive approach[END_REF][START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF][START_REF] Kalevo | Noise reduction techniques for bayer-matrix images[END_REF][START_REF] Zhang | PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras[END_REF] operate under the assumption that DN &DM outperforms DM &DN .

The advantage of DN &DM pipelines is that many excellent denoisers can be applied directly, such as model-based TV [START_REF] Rudin | Nonlinear total variation based noise removal algorithms[END_REF][START_REF] Hintermüller | An adaptive finite element method in l 2 -tvbased image denoising[END_REF][START_REF] Chowdhury | Poisson image denoising based on fractional-order total variation[END_REF][START_REF] Hu | Spatial-frequency domain nonlocal total variation for image denoising[END_REF], non-local [START_REF] Buades | A review of image denoising algorithms, with a new one[END_REF][START_REF] Lebrun | A nonlocal bayesian image denoising algorithm[END_REF][START_REF] Jin | A new nonlocal variational setting for image processing[END_REF][START_REF] Jin | Convergence theorems for the non-local means filter[END_REF][START_REF] Jin | Nonlocal means and optimal weights for noise removal[END_REF], BM3D [START_REF] Dabov | Image denoising by sparse 3-d transformdomain collaborative filtering[END_REF][START_REF] Dabov | Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space[END_REF], low rank [START_REF] Gu | Weighted nuclear norm minimization with application to image denoising[END_REF][START_REF] Guo | Gaussian patch mixture model guided low-rank covariance matrix minimization for image denoising[END_REF] and deep learning-based methods [START_REF] Zhang | Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising[END_REF][START_REF] Zhang | FFDNet: Toward a fast and flexible solution for cnn-based image denoising[END_REF][START_REF] Fang | Multilevel edge features guided network for image denoising[END_REF][START_REF] Guo | Fast, nonlocal and neural: A lightweight high quality solution to image denoising[END_REF], because the statistical nature of the noise is maintained. However, these methods are designed and optimized for grayscale or color images and need to be adapted for application to CFA images [START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF][START_REF] Danielyan | Cross-color bm3d filtering of noisy raw data[END_REF]. Meanwhile, demosaicing algorithms designed on noise-free images can be applied directly after the noise is removed, e.g., [START_REF] Hamilton | Adaptive color plan interpolation in single sensor color electronic camera[END_REF][START_REF] Wu | Temporal color video demosaicking via motion estimation and data fusion[END_REF][START_REF] Mairal | Non-local sparse models for image restoration[END_REF][START_REF] Pekkucuksen | Gradient based threshold free color filter array interpolation[END_REF][START_REF] Buades | Self-similarity driven demosaicking[END_REF][START_REF] Zhang | Color demosaicking by local directional interpolation and nonlocal adaptive thresholding[END_REF][START_REF] Kiku | Residual interpolation for color image demosaicking[END_REF][START_REF] Liang | Wavelet frame based color image demosaicing[END_REF][START_REF] Kiku | Minimized-laplacian residual interpolation for color image demosaicking[END_REF][START_REF] Kiku | Beyond color difference: Residual interpolation for color image demosaicking[END_REF][START_REF] Wu | Demosaicing based on directional difference regression and efficient regression priors[END_REF][START_REF] Tan | Color image demosaicking via deep residual learning[END_REF][START_REF] Jin | A Mathematical Analysis and Implementation of Residual Interpolation Demosaicking Algorithms[END_REF].

For example Park et al. [START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF] consider the classic Hamilton-Adams (HA) [START_REF] Hamilton | Adaptive color plan interpolation in single sensor color electronic camera[END_REF] and a frequency-domain algorithm [START_REF] Dubois | Frequency-domain methods for demosaicking of bayer-sampled color images[END_REF] for demosaicing, combined with two denoising methods, BLS-GSM [START_REF] Portilla | Image denoising using scale mixtures of gaussians in the wavelet domain[END_REF] and CBM3D [START_REF] Dabov | Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space[END_REF]. This combination raises the question of adapting BM3D to a CFA. To do so, the authors first transform the noisy CFA image into the half-size 4-channel image formed by joining the four observed raw values (R,G,G,B) in each four pixel block, then remove noise channel by channel via BM3D [START_REF] Dabov | Image denoising by sparse 3-d transformdomain collaborative filtering[END_REF], finally get the denoised CFA image by the inverse color transform. However, this leads to a checkerboard effect that becomes more noticeable for higher noise levels. Similarly, BM3D-CFA [START_REF] Danielyan | Cross-color bm3d filtering of noisy raw data[END_REF] removes noise directly from the CFA color array, which builds 3D blocks from the same CFA configuration. BM3D-CFA was considered to be a systematic improvement method over [START_REF] Zhang | PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras[END_REF], in which the method [START_REF] Zhang | Color demosaicking via directional linear minimum mean square-error estimation[END_REF] was used as demosaicing method for their comparison of the result after demosaicing. Analogously, [START_REF] Chatterjee | Noise suppression in low-light images through joint denoising and demosaicing[END_REF] adjusted NL-means [START_REF] Buades | A review of image denoising algorithms, with a new one[END_REF] to the CFA image. Zhang et al. [START_REF] Zhang | Joint denoising and demosaicking of noisy cfa images based on inter-color correlation[END_REF] uses a filter [START_REF] Alleysson | Linear demosaicing inspired by the human visual system[END_REF] to extract the luminance of the CFA image. The authors of [START_REF] Zhang | PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras[END_REF] proposed a PCA-based CFA denoising method that makes full use of spatial and spectral correlation. In [START_REF] Patil | Poisson noise removal for image demosaicing[END_REF], Patil and Rajwade remove Poisson noise from CFA images using dictionary learning.

In general, the classical denoising algorithms (such as BM3D, NL-means) can all be adapted to accommodate CFA image denoising in the DN &DM strategy. Several of them [START_REF] Paliy | Demosaicing of noisy data: spatially adaptive approach[END_REF][START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF][START_REF] Kalevo | Noise reduction techniques for bayer-matrix images[END_REF][START_REF] Zhang | PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras[END_REF] address this realistic case by processing the noisy CFA images as a half-size 4-channel color image (with one red, two green and one blue channels) and then apply a multichannel denoising algorithm to it. Albeit the DN &DM pipeline maintains the independent and identically distributed property of the white Gaussian noise (Poisson noise can be transformed to Gaussian noise by the classical Anscombe transform [START_REF] Anscombe | The transformation of poisson, binomial and negative-binomial data[END_REF]), the disadvantage is the reduced resolution of the image (half size), which leads to loss of image detail after denoising. Another issue is that it does not take advantage of the relative spatial position of the R, G, and B pixels due to the separation of the image into four independent channels (R,G,G,B) during denoising, resulting in the color distortion problem. Meanwhile, since G is separated into two independent G channels, the difference between the two G channels after denoising causes checkerboard artifacts.

The DM &DN pipeline was considered for better image detail preservation and to avoid checkerboard artifacts. Unfortunately, there is not many literatures on such pipelines. This is due to the strong spatial and chromatic correlation of the image noise after demosaicing. These correlations are generated by the demosaicing algorithm and are difficult to be modeled, which is detrimental to model-based denoising algorithms. Condat made an attempt in [START_REF] Condat | A simple, fast and efficient approach to denoisaicking: Joint demosaicking and denoising[END_REF], where he first performed demosaicing and then projected the noise into the luminance channel of the reconstructed image and then denoised it based on the grayscale image. The idea was then further refined in [START_REF] Condat | Joint demosaicking and denoising by total variation minimization[END_REF][START_REF] Condat | A generic proximal algorithm for convex optimization-application to total variation minimization[END_REF]. This approach is similar to ours, but we will give a more elaborate theoretical explanation.

To avoid the accumulation of errors caused by the pipeline order, many researchers have proposed to perform a joint demosaicing and denoising [START_REF] Hirakawa | Joint demosaicing and denoising[END_REF][START_REF] Khashabi | Joint demosaicing and denoising via learned nonparametric random fields[END_REF][START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF]. With the popularity of deep learning, joint demosaicing denoising has gained great resolution and excellent performance. By constructing a large number of pairs of simulated data, joint demosaicing and denoising models can be readily trained. Algorithms based on convolutional neural networks (CNNs), such as [START_REF] Syu | Learning deep convolutional networks for demosaicing[END_REF], exhibit performance far exceeding those of handcrafted algorithms [START_REF] Monno | Adaptive residual interpolation for color and multispectral image demosaicking[END_REF]. After [START_REF] Khashabi | Joint demosaicing and denoising via learned nonparametric random fields[END_REF] introduced the first machine learning-based joint demosaicing and denoising method, Gharbi et al. [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF] proposed the first deep learning model. Subsequently, a number of algorithms based on deep learning (such as [START_REF] Dong | Joint demosaicing and denoising with perceptual optimization on a generative adversarial network[END_REF][START_REF] Kokkinos | Iterative joint image demosaicking and denoising using a residual denoising network[END_REF][START_REF] Elgendy | Low-light demosaicking and denoising for small pixels using learned frequency selection[END_REF][START_REF] Liu | Joint demosaicing and denoising with self guidance[END_REF][START_REF] Guo | Joint demosaicking and denoising benefits from a two-stage training strategy[END_REF]) have been proposed. An unsupervised "mosaic-to-mosaic" training strategy for joint demosaicing and denoising was introduced by Ehret et al. [START_REF] Ehret | Joint demosaicking and denoising by fine-tuning of bursts of raw images[END_REF]. In [START_REF] Guo | Joint denoising and demosaicking with green channel prior for real-world burst images[END_REF], Guo et al. focused on joint demosaicing and denoising of real-world burst images. Further, Xing et al. [START_REF] Xing | End-to-end learning for joint image demosaicing, denoising and super-resolution[END_REF] discussed end-to-end joint demosaicing, denoising and super-resolution. In the face of increasing network size and memory consumption, [START_REF] Guan | Memory-efficient deformable convolution based joint denoising and demosaicing for uhd images[END_REF] proposed memory efficient joint demosaicing denoising for Ultra High Definition images. The deep learning-based algorithms mentioned above achieve state-of-the-art performance, but suffer from a common problem of increasingly large network size and high computational complexity. This problem makes deploying these algorithms to devices, especially in low-power or portable devices, difficult to implement. Also, since deep learning algorithms rely on training, generalization issues might arise. For instance, if the noise range used during training is exceeded, or if the image is out of domain, the results might be significantly inferior to those obtained on a testing set. We have briefly summarized the advantages and drawbacks of the three pipelines in Table 1.

In this paper, we address the problem of combining optimally and adapting stateof-the-art demosaicing and denoising algorithms. A preliminary version of this study appeared in [START_REF] Jin | A review of an old dilemma: Demosaicking first, or denoising first?[END_REF]. There, we presented evidence showing that by demosaicking first and then denoising with a higher noise level (denoted DM &1.5DN schemes) yields substantially improved result compared with the classic configurations. This paper extends considerably that preliminary work. In particular, we conduct thorough experiments and develop the arguments to confirm and to extend our conclusions. We first establish a model to optimize the denoising and demosaicing pipeline and use the black box optimizer CMA-ES [START_REF] Hansen | Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation[END_REF] to solve the optimization problem. The optimal results indicate that the DM &1.5DN scheme can get almost the same result as the CMA-ES optimum with a CPSNR value difference ≤ 0.08 dB when σ ≤ 20 and performs much better than DN &DM and DM &DN schemes. Then, we theoretically analyze the statistical properties of demosaiced noise and explain the reason why the DM &1.5DN scheme works well. A series of experiments leads us to conclude that the DM &1.5DN scheme is always superior to the DN &DM and DM &DN ones. For large noise, the best scheme is more complex and has three stages, but we shall show that the DM &1.5DN scheme still is competitive. Our conclusions are different and actually opposite to those of [START_REF] Paliy | Demosaicing of noisy data: spatially adaptive approach[END_REF][START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF][START_REF] Kalevo | Noise reduction techniques for bayer-matrix images[END_REF][START_REF] Zhang | PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras[END_REF]. The advantages of DM &1.5DN scheme seem to be linked to the fact that this scheme does not handle half size 4-channels color image; it therefore uses the classic denoising methods directly on a full resolution color image; this results in more details being preserved and avoids checkerboard artifacts or loss of details. These conclusions also provide theoretical support for real sRGB image denoising [START_REF] Guo | Toward convolutional blind denoising of real photographs[END_REF] which removes noise from full color images after demosaicing. The fact that DM &1.5DN schemes improve on the results of raw image denoising will be verified by experiments carried out on two benchmarks, the Smartphone Image Denoising Dataset (SIDD) [START_REF] Abdelhamed | A high-quality denoising dataset for smartphone cameras[END_REF] and the Darmstadt Noise Dataset (DND) [START_REF] Plötz | Benchmarking denoising algorithms with real photographs[END_REF].

The rest of this paper is structured as follows. In Section 2 we discuss how to apply demosaicing followed by denoising to CFA images. In Section 3, the black box optimizer CMA-ES is used to find the most general 3-step strategy. The results confirm the preference for DM &DN schemes in presence of moderate noise, and lead to a refinement for high noise levels with an DN &DM &DN scheme. In Section 4, we are led to define and analyze the statistical properties of the demosaicing residual noise in RGB and in a transformed space that decorrelates the color channels. Then, using these statistical properties, we find experimentally the appropriate noise level that must be used for the denoising method after demosaicing in a DM &DN scheme. Section 5 compares our strategy with other state-of-the-art ones on simulated and real image datasets. Section 6 concludes.

2. The demosaicing and denoising pipeline. The denoising and demosaicing pipeline consists in solving the ill-posed problem

v = Bayer(u) + ϵ, (1)
where v ∈ R m×n×3 is the observed noisy mosaicked image, Bayer is the Bayer color filter, u = (R, G, B) ∈ R m×n×3 is the latent ground truth color image and ϵ is Gaussian noise with zero mean and standard deviation σ. As stated in the introduction, we will consider the problem of combining demosaicing and denoising, i.e. which one should be executed first? This brings us to two main pipelines: DM &DN (demosaicing then denoising), DN &DM (denoising then demosaicing).

In [START_REF] Jin | A review of an old dilemma: Demosaicking first, or denoising first?[END_REF], we reached the preliminary conclusion: demosaicing should be executed with higher priority and subsequent denoising needs to be adjusted. In the next section we will propose to consolidate (and partly modify) this conclusions by optimizing freely a 3-step procedure. Let σ 1 and σ 2 be the noise level hyperparameters of DN &DM and DM &DN respectively. The restored image can be evaluated by subjective criteria such as visual quality and by objective criteria such as the color signal-to-noise ratio (CPSNR) [START_REF] Alleysson | Linear demosaicing inspired by the human visual system[END_REF], defined by

CPSNR( u) = 10 log 10 255 2 MSE( u) , with (2)
MSE( u) = 1 m × n × 3 ∥ u -u∥ 2 F
, where ∥ • ∥ F is the Frobenius norm, u denotes the ground truth image and u is the estimated color image.

Park et al. [START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF] argued that demosaicing introduces chromatic and spatial correlations to the noise, which is no longer i.i.d. white Gaussian and therefore harder to model and eliminate. In [START_REF] Kalevo | Noise reduction techniques for bayer-matrix images[END_REF] the authors argue that DN &DM schemes with a proper parameter are more efficient than DM &DN schemes. Figure 1 (d) shows an example where a noisy CFA image with noise of standard deviation σ was first demosaiced by RCNN [START_REF] Tan | Color image demosaicking via deep residual learning[END_REF] and then restored by CBM3D [START_REF] Dabov | Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space[END_REF] assuming a noise parameter σ 2 = σ. The output of CBM3D with σ 2 = σ has a strong residual noise. A similar behavior is also observed with other image denoising algorithms such as nlBayes [START_REF] Jin | A review of an old dilemma: Demosaicking first, or denoising first?[END_REF]. Based on this argument several papers [START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF][START_REF] Zhang | PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras[END_REF][START_REF] Akiyama | Pseudo four-channel image denoising for noisy cfa raw data[END_REF][START_REF] Lee | Denoising algorithm for cfa image sensors considering interchannel correlation[END_REF] propose raw CFA denoising methods applicable before demosaicing.

Other denoising methods that are not explicitly designed to handle raw CFA images (such as CBM3D and nlBayes) can also be adapted to noisy CFA images by rearranging the CFA image into a half-size four-channels image [START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF], or two halfsize three-channel images as shown in Figure 2. In our comparative experiments, CBM3D will be used to process CFA images, which is the scheme in Figure 2, we will denote this method as cfaBM3D. In the case of splitting the raw image into two half-size 3-channel images (see Figure 2), both images are denoised independently and the denoised pixels are recombined. Each half-size image contributes one green pixel to the denoised CFA image, while the red and blue pixels are averaged. Despite the DN &DM pipeline effectively eliminates noise, it is not good at preserving details and produces artifacts such as checkerboard effect. Indeed, due to the rearrangement of the CFA pixels, much image detail is lost in the image after applying an DN &DM scheme. In addition, this procedure introduces visible checkerboard artifacts for noise levels σ > 10. These artifacts can be observed in Figure 1 (c). To address this last issue, Danielyan et al. [START_REF] Danielyan | Cross-color bm3d filtering of noisy raw data[END_REF] proposed BM3D-CFA, which amounts to denoise four different mosaics of the same image before aggregating the four values obtained for each pixel. In practice, we observed that BM3D-CFA and the cfaBM3D method described above attain very similar results. The main difference between the two comes with the execution time, as for cfaBM3D a fast GPU implementation is available [START_REF] Davy | Gpu acceleration of nl-means, bm3d and vbm3d[END_REF]. Depending on the experiment we will use one or the other.

Jin et al. [START_REF] Jin | A review of an old dilemma: Demosaicking first, or denoising first?[END_REF] revised the DM &DN pipeline and observed that a very simple modification of the noise parameter of the denoiser DN coped with the structure of demosaiced noise, and led to efficient denoising after demosaicing, i.e. a DM &1.5DN pipeline. This allows for a better preservation of fine structure often smoothed by the DN &DM schemes, and checkerboard artifacts are no longer observed (see Figure 1 (e)). In terms of quality and speed, demosaicing DM can be done by a fast algorithm RCNN [START_REF] Tan | Color image demosaicking via deep residual learning[END_REF] followed by CBM3D denoising 1.5DN , namely CBM3D applied with a noise parameter equal to σ 2 = 1.5σ.

Figure 1 also illustrates that DN &DM has better CPSNR than DM &DN . However, the performance of DM &1.5DN pipeline is much superior to both DM &DN and DN &DM . Is DM &1.5DN pipeline the optimal one? In Section 3, we will explore a more generic optimal pipeline of denoising and demosaicing to confirm this optimality for moderate noise, and a near optimality for large noise. In Section 4, based on the analysis of demosaiced noise we shall seek an explanation of the efficiency of DM &1.5DN .

3. Pipeline optimization and analysis. In order to arrive at a rigorous decision in a more general framework, we designed a generic DN 1 &DM &DN 2 pipeline. The structure of the pipeline is illustrated in Figure 3. This pipeline allows for an arbitrary order between DN and DM and sets free their parameters. It has two denoisers and four hyperparameters. The two denoisers are a CFA denoiser DN 1 (see Figure 2) and a full color image denoiser DN 2 , which respectively remove noise before and after demosaicing. The four hyperparameters are α (that controls the weight of CFA denoising), β (that controls the weight of color denoising), σ 1 (the noise standard deviation of the CFA denoiser), σ 2 (the noise standard deviation of the color denoiser). The results of the pipeline are visualised in Figure 4. The final result of the pipeline is given by

u = βDN 2 (DM ( v), σ 2 ) + (1 -β)DM ( v), (3)
where

v = αDN 1 (v, σ 1 ) + (1 -α)v. It follows that if α = 1, β = 0, σ 1 = σ and σ 2 = 0, then v = DN (v) and u = DM (DN (v)), i.e. the pipeline is DN &DM ; if α = 0, β = 1, σ 1 = 0 and σ 2 = σ, then v = v and u = DN (DM (v)), i.e. the pipeline is DM &DN ; if α = 0, β = 1, σ 1 = 0 and σ 2 = 1.5σ, then v = v and u = DN (DM (v)), i.
e. the pipeline is DM &1.5DN [START_REF] Jin | A review of an old dilemma: Demosaicking first, or denoising first?[END_REF]. Our purpose is, for every noise level σ, to find the optimal values {α * , β where u is defined by (3) and CPSNR is defined in [START_REF] Akiyama | Pseudo four-channel image denoising for noisy cfa raw data[END_REF]. Obviously, problem (4) is non-linear, non-convex and the gradients are not readily available. In order to obtain the optimal solution of (4) (and inspired by [START_REF] Mosleh | Hardware-inthe-loop end-to-end optimization of camera image processing pipelines[END_REF]), we used the black box optimizer CMA-ES [START_REF] Hansen | Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation[END_REF], which is a random search optimizer that is based on evolutionary strategies. Unlike common gradient optimization, CMA-ES does not compute the gradient of the objective function. Only the ranking between candidate solutions is exploited for learning the sample distribution; neither derivatives nor even the function values themselves are required by the method [START_REF] Heris | Implementation of covariance matrix adaptation evolution strategy (cma-es) in matlab[END_REF].

* , σ * 1 , σ * 2 } satisfying {α * , β * , σ * 1 , σ * 2 } = arg max {α,β,σ1,σ2} CPSNR( u), (4)
We carried out experiments with different noise levels (σ = 5, 10, 20, 40, 50, 60) on the images from the Imax [START_REF] Zhang | Color demosaicking by local directional interpolation and nonlocal adaptive thresholding[END_REF] and Kodak [START_REF] Franzen | Kodak lossless true color image suite[END_REF] datasets. In this experiment we used the denoiser with the framework Figure 2 for DN 1 , MLRI for DM and CBM3D [START_REF] Dabov | Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space[END_REF] for DN 2 . For each experiment, {α, β, σ 1 , σ 2 } were initialized randomly. Figure 5 illustrates the evolution of the CPSNR during the optimization with respect to {α, β, σ 1 , σ 2 }. In all cases, the parameters and the CPSNR stabilize after about 60-iterations. The final results are shown in Table 2 along with results of the DN &DM method (cfaBM3D+MLRI)1 , the DM &DN method (MLRI+CBM3D) .0225 to 0.9030 while β remains always larger than 0.9. This means that applying denoising before demosaicing is not important for low noise levels, but becomes necessary when σ increases, while applying denoising after demosaicing is always favorable, but with a little smaller denoising parameters.

When the noise level is high, the CPSNR of DM &1.5DN is 0.3 to 0.4 dB below the optimal value obtained by the DN 1 &DM &DN 2 pipeline. however, this requires almost doubling the computational complexity due to denoising. Therefore, by trading-off image quality and computational cost, the simplified DM &1.5DN pipeline remains a good option and it is almost optimal for moderate noise. For this reason, we shall explore in detail this pipeline and the reasons of its near optimality in the next section.

4. Analysis of DM &1.5DN . As we saw in Section 3, The result of the DM &1.5DN pipeline is almost equal to the result of the optimal DN 1 &DM &DN 2 pipeline and much better than the DN &DM pipeline for all noise levels. The fact that a DM &1.5DN pipeline surpasses than a DN &DM scheme is surprising, considering that after demosaicing the noise is no longer white. Indeed, chromatic and spatial correlations have been introduced by the demosaicing, while the applied denoiser was conceived for white noise. This apparent paradox leads us to analyze the behavior of demosaiced noise. Figure 6 illustrates the above definition. The demosaiced noise is nothing but the difference between the demosaiced version of a noisy image and its underlying ground truth. The demosaiced noise of column (d) is (visually) not significantly higher than the white noise of column (b), but it is clearly no longer white, due to the introduction of chromaticity and spatial correlations. The properties of the demosaiced noise depend on the demosaicing algorithm, as developed in [START_REF] Jin | A review of an old dilemma: Demosaicking first, or denoising first?[END_REF]. This paper compares DM &1.5DN pipelines composed of seven different state-of-the-art demosaicing algorithms (such as HA [START_REF] Hamilton | Adaptive color plan interpolation in single sensor color electronic camera[END_REF], GBTF [START_REF] Pekkucuksen | Gradient based threshold free color filter array interpolation[END_REF], RI [START_REF] Kiku | Residual interpolation for color image demosaicking[END_REF] and so on). To understand empirically the right noise model to adopt after demosaicing, and following the conclusions of [START_REF] Jin | A review of an old dilemma: Demosaicking first, or denoising first?[END_REF], we applied CBM3D after demosaicing with a noise parameter σ 2 corresponding to σ multiplied by (1.0, 1.1, • • • , 1.9). These experiments show that the optimal parameter interval is [1.4, 1.7] and that the optimal factor is 1.5.

This surprising result would seem to imply that demosaicing increases noise. But this is not the case, as illustrated in Table 3, which gives the noise standard deviation estimated as the mean RMSE of the demosaiced images from the Imax [START_REF] Zhang | Color demosaicking by local directional interpolation and nonlocal adaptive thresholding[END_REF] dataset with different noise levels. For low noise (σ = 1) the large demosaicing error of about 4 clearly is caused by the demosaicing itself. However, for σ > 10 the RMSE of the demosaiced image tends to be roughly equal to 3/4 of the initial noise standard deviation. In short, as expected from an interpolation algorithm, demosaicing (slightly) decreases the noise standard deviation. This is also consistent with the visual results observed in Figure 6.

At first sight, this 3/4 factor contradicts the observation that denoising with a parameter σ 2 = 1.5σ yields better results. This leads us to further analyze the structure of the demosaiced residual noise. To that aim, we applied an orthonormal 4 for quantitative results.

Karhunen-Loeve transform to the residual noise to maximally decorrelate the color channels [START_REF] Malvar | High-quality linear interpolation for demosaicing of bayer-patterned color images[END_REF][START_REF] Ohta | Color information for region segmentation[END_REF]. This transform is commonly used in denoising algorithms [START_REF] Lebrun | Secrets of image denoising cuisine[END_REF] such as CBM3D [START_REF] Dabov | Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space[END_REF]. Here, we used a transform (R,

G, B) → (Y, C 1 , C 2 ), in which the luminance direction is Y = R+G+B √ 3
and the orthogonal vectors C 1 and C 2 are arbitrarily chosen as in [START_REF] Kalevo | Noise reduction techniques for bayer-matrix images[END_REF], which is defined as

  Y C 1 C 2   =   1/ √ 3 1/ √ 3 1/ √ 3 1/ √ 2 0 -1/ √ 2 1/ √ 6 -2/ √ 6 1/ √ 6     R G B   . (5)
The color distortion caused by denoising in the YC 1 C 2 space is much less than that in the RGB space, and this transformation does not change the properties of independent identically distributed noise. This explains why it is generally used for color image denoising. We further analyze the properties of residual noise in the YC 1 C 2 color space.

From Figure 7 one can see that the AWG noise is isotropic whereas the demosaiced noise is not isotropic anymore in the RGB space. The noise is elongated in the brightness direction Y = R+G+B √ 3 , and compressed in other directions. Furthermore, the noise becomes blurred after demosaicking. This indicates that the Table 4. Noise intensity. Variance and covariance of (R, G, B) and (Y, C 1 , C 2 ) between pixels (i, j) and (i + s, j + t), s, t = 0, 1, 2 first for AWGN (a) with standard deviation σ = 20, then for its demosaiced versions by HA (b), RI (c), MLRI (d) and RCNN (e).

(i,j )(i,j +1)(i,j +2)(i+1,j )(i+1,j +1)(i+1,j +2)(i+2,j )(i+2,j +1)(i+2,j +2) (i,j )(i,j +1)(i,j +2)(i+1,j )(i+1,j +1)(i+1,j +2)(i+2,j )(i+2,j +1)(i+2,j +2) R 361.4 128. [START_REF] Anscombe | The transformation of poisson, binomial and negative-binomial data[END_REF] (i,j )(i,j +1)(i,j +2)(i+1,j )(i+1,j +1)(i+1,j +2)(i+2,j )(i+2,j +1)(i+2,j +2) R 359. [START_REF] Chen | An efficient statistical method for image noise level estimation[END_REF] demosaiced noise is correlated between adjacent pixels. This is also verified in Table 4 which illustrates the variances and covariances of AWGN and demosaicked noise with σ = 20 both in RGB and YC 1 C 2 spaces. One can observe that the statistical properties of AWG noise remains unchanged while that of demosaicked noise changes obviously after (R, G, B) → (Y, C 1 , C 2 ) transformation. The variance of Y is a growing sequence for the demosaiced noise obtained by increasingly sophisticated demosaicing: 654 for HA, 715 for RI, 772 for MLRI, 972 for RCNN. Table 5. Correlation between pixels. The corresponding correlations of (R, G, B) and (Y, C 1 , C 2 ) between pixels (i, j) and (i + s, j + t), s, t = 0, 1, 2 first for AWGN (a) with standard deviation σ = 20, then for its demosaiced versions by HA (b), RI (c), MLRI (d) and RCNN (e).

R
(i,j )(i,j +1)(i,j +2)(i+1,j )(i+1,j +1)(i+1,j +2)(i+2,j )(i+2,j +1)(i+2,j +2) R
(i,j )(i,j +1)(i,j +2)(i+1,j )(i+1,j +1)(i+1,j +2)(i+2,j )(i+2,j +1)(i+2,j +2) R
(i,j ) (i,j +1) (i,j +2) (i+1,j ) (i+1,j +1)(i+1,j +2) (i+2,j ) (i+2,j +1)(i+2,j +2) R 1.0000 0.0015 0.0010 0.0017 0.0002 0.0018 0.0007 0.0005 0.0021 G 1.0000 0.0012 0.0028 0.0004 0.0007 0.0023 0.0025 0.0016 0.0010 B 1.0000 0.0029 0.0002 0.0013 0.0015 0.0001 0.0047 0.0008 0.0047 Y 1.0000 0.0028 0.0004 0.0007 0.0002 0.0023 0.0005 0.0012 0.0030 C 1 1.0000 0.0003 0.0021 0.0016 0.0007 0.0008 0.0024 0.0011 0.0033 C 2 1.0000 0.0005 0.0045 0.0023 0.0005 0.0025 0.0014 0.0005 0.0005 (a) AWGN Hence, the noise standard deviation on Y has been multiplied by a factor between 1.27 and 1.56. In contrast, the demosaiced noise is reduced in the C 1 and C 2 axes, with its variance passing from 400 for AWGN to 168 and 94 for RI, and even down to 43 and 55 for RCNN. Table 4 also shows that the covariances between adjacent pixels are no longer close to 0 and that the covariances of demosaicked noise is Table 6. Correlation between channels. Covariance (each first row) and corresponding correlation (each second row) of the three color channels (R, G, and B) of the demosaicing noise when the initial CFA white noise satisfies σ = 20. See Figure 7 for an illustration. an almost descending sequence by increasingly sophisticated demosaicing. In order to further analyze the correlation between adjacent pixel noises, the correlation coefficients of adjacent pixel noises are calculated and listed in Table 5. The correlation of AWGN is (almost) 0 due to the independent properties (see Table 5 (a)). However, the demosaiced noise have a strong correlation in (R, G, B) color space. After transformation, the channel correlation of Y decreases significantly and the correlation of C 1 and C 2 increases. These observations lead to a simple conclusion: As the computational complexity increases, the Y component of the demosaiced noise gets closer to white. However, the residual noise on C 1 and C 2 is strongly spatially correlated, it is therefore a low frequency noise, that will require stronger filtering than white noise to be removed. Since image denoising algorithms are guided by the Y component [START_REF] Dabov | Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space[END_REF][START_REF] Lebrun | A nonlocal bayesian image denoising algorithm[END_REF], we can denoise with methods designed for white noise, but with a noise parameter adapted to the increased variance of Y.

(i,j ) (i,j +1) (i,j +2) (i+1,j ) (i+1,j +1)(i+1,j +2) (i+2,j ) (i+2,j +1)(i+2,j +2) R 1.
(i,j ) (i,j +1) (i,j +2) (i+1,j ) (i+1,j +1)(i+1,j +2) (i+2,j ) (i+2,j +1)(i+2,j +2) R 1.
(i,j ) (i,j +1) (i,j +2) (i+1,j ) (i+1,j +1)(i+1,j +2) (i+2,j ) (i+2,j +1)(i+2,j +2) R 1.
R G B R G B R
To understand why the variance of Y is far larger than the AWGN it comes from, let us study in Table 6 the correlation between the three channels (R, G, B) in the demosaiced noise of HA, RI, MLRI and RCNN. We observe a strong (R, G, B) correlation ranging from 0.4 for HA to 0.89 for RCNN, which is caused by the "tendency to grey" of all demosaicing algorithms (see Figures 6 and7). Assuming that the demosaiced noisy pixel components (denoted ϵ R , ϵ G , ϵ B ) have a correlation coefficient close to 1 then we have

Y = ϵ R + ϵ G + ϵ B √ 3 ∼ √ 3 N (0, σ).
This factor of about 1.7 corresponds to the case with maximum correlation. The empirical observation of an optimal factor near 1.5 responds to a lower correlation between the colors. All in all, the analysis of the statistical properties of demosaicked noise explains why the DM &DN scheme with an appropriate parameter σ 2 = 1.5σ performs similarly to the optimal CMA-ES, and is much than DN &DM .

Experimental evaluation.

To evaluate the proposed framework for denoising and demosaicing, we conducted experiments on simulated images and real images separately. The most classic Imax [START_REF] Zhang | Color demosaicking by local directional interpolation and nonlocal adaptive thresholding[END_REF] and Kodak [START_REF] Franzen | Kodak lossless true color image suite[END_REF] datasets were selected for the simulated images. To verify the effect on real raw images, we also evaluated it on the SIDD dataset [START_REF] Abdelhamed | A high-quality denoising dataset for smartphone cameras[END_REF] and on the DND [START_REF] Plötz | Benchmarking denoising algorithms with real photographs[END_REF] benchmark. The former comes with ground truth acquisitions, while the latter allows to evaluate the results by submitting them to the benchmark website. We compared nine different pipelines, namely:

• Best performing DN &DM and DM &1.5DN pipelines built by RCNN [START_REF] Tan | Color image demosaicking via deep residual learning[END_REF] and cfaBM3D or CBM3D [START_REF] Dabov | Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space[END_REF]. • Low cost DN &DM , DM &1.5DN and CMA-ES pipelines built by MLRI [START_REF] Kiku | Minimized-laplacian residual interpolation for color image demosaicking[END_REF] and cfaBM3D or CBM3D [START_REF] Dabov | Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space[END_REF]. • The CFA denoising framework proposed by Park et al. in [START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF], which effectively compresses the signal energy by using a color representation obtained by principal component analysis of the Kodak dataset, and then removes the noise in each channel by BM3D. We combined this framework with RCNN [START_REF] Tan | Color image demosaicking via deep residual learning[END_REF]. • The PCA-CFA filter proposed in [START_REF] Zhang | PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras[END_REF] uses principal component analysis (PCA) and spatial and spectral correlation of images to preserve color edges and details. We combined it with DLMM demosaicing [START_REF] Zhang | Color demosaicking via directional linear minimum mean square-error estimation[END_REF] and RCNN demosaicing [START_REF] Tan | Color image demosaicking via deep residual learning[END_REF].

• Since 2016, solving joint demosaicing denoising has typically used deep learning. As a reference, we included JCNN [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF][START_REF] Ehret | A study of two CNN demosaicking algorithms[END_REF], which is one of the classical deep learning algorithms for this problem, for comparison. It is important to emphasize that it was trained on noise standard deviations σ ≤ 20 only. Table 7 shows that RCNN+1.5CBM3D obtains the optimum on average. It comes to no surprise that JCNN [START_REF] Gharbi | Deep joint demosaicking and denoising[END_REF][START_REF] Ehret | A study of two CNN demosaicking algorithms[END_REF] performs slightly better than the other methods on the Imax dataset. Table 8 shows that the DM &1.5DN method RCNN + 1.5CBM3D yields the best results on the Kodak dataset. And when the noise increases, the 'low-cost' MLRI+1.5CBM3D also achieves impressive results. However, it is restricted to a limited range of noise levels and cannot handle the noise levels outside the training range. Furthermore, it requires much more memory and computation. In summary, DM &1.5DN methods are more robust and have a better performance than cfaBM3D+RCNN. All DM &1.5DN methods outperform the DN &DM methods Park+RCNN [START_REF] Park | A case for denoising before demosaicking color filter array data[END_REF], PCA+DLMM [START_REF] Zhang | PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras[END_REF] and PCA+RCNN [START_REF] Zhang | PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras[END_REF].

We now examine the visual quality of restored images. Figures 8-10 compare the visual quality obtained by the main discussed methods. Key parts of images were zoomed-in for a better view. From the upper-left extract of Figure 8, we can see that textures are well restored by RCNN+1.5CBM3D and MLRI+1.5CBM3D, while they are blurred the cfaBM3D+RCNN and destroyed by JCNN. In the lowerleft extract, the girl's hairs are oversmoothed by cfaBM3D+RCNN and JCNN but are well preserved by our proposed method. In the upper-left and lower-left corner of Figure 9, cfaBM3D+RCNN oversmooths the details and JCNN introduces some artifacts at the window and oversmooths the door. Instead, RCNN+1.5CBM3D preserves the details and does not introduce artifacts. The zoomed-in parts of Figure 10 show that JCNN and cfaBM3D+RCNN introduce checkerboard artifacts while methods based on the DM &1.5DN scheme do not. The advantage of our proposed approach becomes more obvious when dealing with high noise. There are severe checkerboard artifacts in the images restored by cfaBM3D+MLRI and cfaBM3D+RCNN (see in the bottom left-hand corner of the image of Figure 11), and the details are oversmoothed (see in the upper left corner of the image of Figure 11), while our proposed approach not only avoids checkerboard artifacts, but also retains the details. The image restored with JCNN is very noisy because JCNN was not trained beyond σ = 20. As a rule of thumb, the DM &DN scheme with an appropriate parameter (namely DM &1.5DN ) outperforms the competition in terms of visual quality. This is due to the fact that it efficiently uses spatial and spectral image characteristics to remove noise, preserve edges and fine detail. Indeed, contrary to the DN &DM schemes, DM &1.5DN does not reduce the resolution of the noisy image. Using a DN &DM scheme ends up over-smoothing the result. A comparison of CPSNRs and visual quality on these simulated examples leads to conclude that the DM &1.5DN scheme is indeed much more robust and better performing than the DN &DM scheme. 5.2. Evaluation of DN &DM and DM &1.5DN strategies on real image datasets. In order to prove the advantage of a DM &1.5DN strategy on real images, we evaluated its application to the real sRGB images taken from the SIDD dataset [START_REF] Abdelhamed | A high-quality denoising dataset for smartphone cameras[END_REF]. In this dataset, the noisy sRGB images and their corresponding ground the images is in the range σ ∈ [3. 28, 38.12], and the noise level of most of the images (≥ 93.75%) is no higher than 20. This justifies the choice of DM &1.5DN . Table 9 shows the CPSNR and estimated noise levels of images generated by different schemes on the SIDD dataset. We list them separately by phone model. It can be seen from Table 9 that the DM &1.5DN solution is more competitive than the DN &DM solution in terms of CPSNR, with an average 0.60 dB gain. This is consistent with the previous results on the simulated data. Figure 12 shows the visual quality of both strategies. JCNN is not competitive on the SIDD dataset, because it was not trained on this dataset. This also shows that our proposed scheme has better robustness and adaptability than JCNN. The DM &1.5DN scheme keeps more image details than others.

In a word, the DM &1.5DN scheme clearly outperforms DN &DM in visual quality and numerical results for both simulated data and real data. Our results also provide theoretical support for real sRGB image denoising which removes noise from full color images after demosaicing. The next section addresses raw image denoising.

5.3.

The DM &1.5DN strategy for raw image denoising. We applied the DM &1.5DN scheme to raw image denoising. To that aim, we defined the pipeline shown in Figure 13. We considered two pipeline variants: with and without variance stabilizing transform. In the first case, a variance stabilizing transformation was used to transform the raw image noise into approximate Gaussian noise, and the noise level in each image was then estimated by the method [START_REF] Chen | An efficient statistical method for image noise level estimation[END_REF]. In the second case, we applied the noise estimation method [START_REF] Chen | An efficient statistical method for image noise level estimation[END_REF] directly on the original noise images. Table 10 shows the results of the DM &1.5DN scheme on the raw images of the SIDD dataset [START_REF] Abdelhamed | A high-quality denoising dataset for smartphone cameras[END_REF]. Note that applying the VST leads to slightly better results in almost all cases. RCNN underperforms when handling raw data, because its training data is sRGB data. MLRI is a traditional interpolation algorithm, which is not affected by different color spaces and achieves the best results. The estimated noise range for the original noisy images in the SIDD raw image datasets is σ ∈ [0.48, 22.59] and after VST is σ ∈ [0.38, 13.00]. According to Table 2, the results of the CMA-ES optimized scheme and the DM &1.5DN scheme are almost equal when the noise level σ ≤ 20, which justifies the use of DM &1.5DN (more precisely, the noise level of all considered images is always less than 23). Considering the trade-off between reconstruction quality and computational consumption, the DM &1.5DN scheme is more valuable for the considered application.

To further validate the performance of the DM &1.5DN scheme, we compared MLRI+CBM3D and RCNN+CBM3D with TNRD [START_REF] Chen | On learning optimized reaction diffusion processes for effective image restoration[END_REF], EPLL [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF], WNNM [START_REF] Gu | Weighted nuclear norm minimization with application to image denoising[END_REF], BM3D [START_REF] Dabov | Image denoising by sparse 3-d transformdomain collaborative filtering[END_REF] and CycleISP [START_REF] Zamir | CycleISP: Real image restoration via improved data synthesis[END_REF] on the SIDD [START_REF] Abdelhamed | A high-quality denoising dataset for smartphone cameras[END_REF] and DND [START_REF] Plötz | Benchmarking denoising algorithms with real photographs[END_REF]

. Denoising results on an image from the DND dataset. We compare the DM &1.5DN scheme (MLRI+CBM3D and RCNN+CBM3D), TNRD [START_REF] Chen | On learning optimized reaction diffusion processes for effective image restoration[END_REF], EPLL [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF], WNNM [START_REF] Gu | Weighted nuclear norm minimization with application to image denoising[END_REF] and BM3D [START_REF] Dabov | Image denoising by sparse 3-d transformdomain collaborative filtering[END_REF] (results as reported on the benchmark website).

use case for DM &1.5DN . The relevant results are shown in Table 11, and more detailed results can be found on the SIDD2 and DND3 websites. The CycleISP result is better on DND than our best proposed scheme MLRI+CBM3D, but not 14). Third, MLRI and CBM3D can be accelerated without performance loss on dedicated architectures while the computational weight of a CNN is hardly reducible. Although the DM &1.5DN scheme falls short of state-of-the-art deep learning raw image denoising methods such as CycleISP [START_REF] Zamir | CycleISP: Real image restoration via improved data synthesis[END_REF], our proposed lightweight scheme is still the best among traditional algorithms and it even outperforms some deep learning algorithms (see the DND benchmark website). Compared to the computational resources consumed by deep learning methods, our proposed scheme is computationally very competitive. Figure 14 shows the comparison of the visual quality of traditional algorithms on raw image denoising. Our scheme keeps more details, introduces fewer color artifacts than other traditional algorithms and avoids checkerboard artifacts. With a lightweight demosaicker, BM3D obviously improves on raw image denoising with an average gain of 3.91 dB for SIDD, 0.99 dB for DND and 0.61 dB for DND with VST. As a result, we can conclude that the DM &1.5DN scheme is very effective for raw image denoising. 5.4. Time consumption and generalizability. We examined the runtimes of three strategies and evaluated the generalizability of the CMA-ES scheme, aiming to achieve a balance between good performance and reasonable runtimes. We limited our comparison to traditional algorithms, as deep learning algorithms require long computing times on CPUs. Table 12 shows the running times of the three strategies on a PC with an Intel Core i7-9750H 2.60GHz CPU and 16GB memory. As the table demonstrates, the demosaicing algorithm has a negligible runtime, while the majority of the computational time is spent on denoising. The computation time of DN &DM is half that of DM &1.5DN , because DN &DM processes two halfsize images, which is exactly half the size of the full-color images processed by DM &1.5DN . In terms of the trade-off between time consumption and performance, DM &1.5DN is the optimal choice, particularly for moderate levels of noise (σ ≤ 20, as described in Section 5.3). However, for high noise scenes, the DN 1 &DM &DN 2 pipeline may be the best option for achieving optimal performance.

We now turn our attention to the generalization of the CMA-ES optimization parameters, which requires a large number of calculations, making the optimization process time-consuming. One critical aspect is the independence of the parameters Table 13. Generalizability of CMA-ES optimal parameters to different noise levels. Evaluation of noise levels with σ = 50 proximity (selected as 46 to 54) using two generalization schemes. from the dataset. This issue arises implicitly in the previous discussion. In Section 3, we employed the Imax dataset for the CMA-ES optimization, whereas the parameters were applied directly to the Kodak dataset in the comparison (see Tables 2 and8). As demonstrated in these tables, the CMA-ES optimal parameters remain consistent when applied to the Kodak dataset, which leads to the conclusion that the CMA-ES optimization parameters exhibit good generalization across datasets. Another crucial aspect is the generalization to different noise levels. Given that it is impractical to train optimal parameters each time for real-world applications, it is essential to discuss what to do when the noise level does not match the level of optimal parameters. We propose two schemes:

• Image transformation, where the image is transformed to the nearest noise level using the corresponding optimal parameters α, β, σ 1 , σ 2 , namely x σ * σ and its inverse y σ σ * , where x is the noisy image, y is the reconstructed image, σ * is the actual noise level, and σ is the nearest noise level with known optimal parameters; • σ transformation, where the optimal parameter α, β for the nearest noise level is directly used, and the parameters σ 1 and σ 2 are transformed by σ * 1 = σ1 σ σ * and σ * 2 = σ2 σ σ * , where σ * is the actual noise level, and σ is the nearest noise level with known optimal parameters. We evaluated the how both schemes generalize around σ = 50 (selected as 46 to 54). The corresponding results are presented in Table 13. As shown in the table, both schemes outperform the DN &DM and DM &1.5DN strategies, indicating the generality of the CMA-ES optimization parameters over a range without the need for repeated optimization.

From Table 12, it is apparent that the denoising stage is responsible for the majority of the time consumption. Therefore, it is advisable to use a fast algorithm, such as the BM3D algorithm implemented on the GPU [START_REF] Davy | Gpu acceleration of nl-means, bm3d and vbm3d[END_REF] when using the CMA-ES algorithm to obtain optimal parameters. 6. Conclusion. This paper established a model to optimize the denoising and demosaicing pipeline. The optimal pipeline (obtained by CMA-ES) is a DN 1 &DM &DN 2 scheme with appropriate parameters and DM &1.5DN is almost equal to the optimal one when σ ≤ 20. Our best performing combination in terms of quality and speed is a DM &1.5DN scheme for two reasons: the DN 1 &DM &DN 2 scheme gets the best result, but it takes twice as many calculations as DM &1.5DN ; as discussed in Section 5.3, in most cases, the noise level for raw images is less than 20. Experiments show a considerable gain. The results of the DM &1.5DN scheme show a 0.5 to 1 dB gain, when compared with the best DN &DM strategy. These conclusions apply for moderate noise (σ ≤ 20) but remain valid for high noise, where we nevertheless found a slight improvement of about 0.3 dB for a twice more complex pipeline DN 1 &DM &DN 2 with two denoising steps. We also gave a detailed theoretical explanation of why the DM &1.5DN scheme is superior to the DN &DM scheme.

We also saw that, unsurprisingly, heavy weight learning-based joint demosaicing and denoising achieves the best performance. However, the above conclusions are still crucial for practical light weight and domain independent application scenarios. They might also inspire the design and training of deep learning algorithms.

Figure 1 .

 1 Figure 1. Image details at σ = 20. The lower row is the reconstructed image, and the upper row is the difference between the reconstructed image and ground truth. DN : cfaBM3D or CBM3D denoising; DM : RCNN demosaicing. 1.5DN means that if the noise level is σ, the input noise level parameter of denoising method DN is σ 2 = 1.5σ.

Figure 2 .

 2 Figure 2. The framework used for denoising before demosaicing using an RGB denoiser. The Bayer CFA image is split in two half resolution RGB images, each one with a different green. Both RGB images are denoised independently. Then the pixels of both results are recombined into a denoised Bayer CFA image. The last step consists in applying a demosaicing algorithm.

Figure 3 .

 3 Figure 3. Generic raw image processing pipeline. This pipeline structure allows for an arbitrary order between DN and DM and sets free their parameters. We use the CMA-ES algorithm to optimize the parameter α, β, σ 1 , σ 2 in the pipeline.

Figure 4 .

 4 Figure 4. A visual representation of the process in Figure 3, where the noise level is σ = 60. The parameter are α = 0.90, β = 0.99, σ 1 = 34.50, σ 2 = 54.42. Since β is always close to 1 in the pipeline, the visual difference between Color Denoising and the β linear combination is not significant.

Figure 5 .

 5 Figure 5. Evolution of the result of iterating CMA-ES when optimizing the parameters α, β, σ 1 , σ 2 of the processing pipeline.

Figure 6 .

 6 Figure 6. First row: (a) Ground truth Imax 3, (b) its noisy version, (c) added white noise (σ = 20), (d) demosaiced version of (b) by RCNN, (e) the demosaiced noise, namely the difference (d)-(a). Second and third rows: 50 × 50 extracts from the first row.

Definition 4 . 1 .

 41 Consider a ground truth color image (R, G, B) and its mosaic obtained by keeping only one value of either R, G, B at each pixel, on a fixed Bayer pattern. Assume that white noise with standard deviation σ has been added to the mosaicked image, and that the resulting noisy mosaic has been demosaiced by DM , hence giving a noisy image ( R, G, B). We then call demosaiced noise the difference ( R -R, G -G, B -B).

Figure 7 .

 7 Figure 7. AWGN image and demosaicing noise with standard deviation σ = 20 for respectively HA, MLRI, RCNN. Last row: the color spaces (in standard (R,G,B) Cartesian coordinates) of each noise, presented in their projection with maximal area. As expected, the AWG color space is isotropic, while the color space after demosaicing is elongated in the luminance direction Y and squeezed in the others. This amounts to an increased noise standard deviation for Y after demosaicing, and less noise in the chromatic directions. See tableTable4for quantitative results.

5. 1 .

 1 Evaluation of DN &DM and DM &1.5DN strategies on simulated images. All Imax and Kodak images were corrupted by AWGN with standard deviations σ = 5, 10, 20, 40, 50, 60.

Figure 8 .

 8 Figure 8. Demosaicing and denoising results on an image from the Kodak dataset with σ = 20. We compare the two schemes of DN &DM , cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of DM &1.5DN , MLRI+CBM3D and RCNN+CBM3D, and the MLRI+CBM3D schemes optimized by the CMA-ES algorithm. As a reference we also include the result of JCNN, a joint CNN method.

Figure 9 .

 9 Figure 9. Demosaicing and denoising results on an image from the Kodak dataset with σ = 10. We compare the two schemes of DN &DM , cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of DM &1.5DN , MLRI+CBM3D and RCNN+CBM3D, and the MLRI+CBM3D schemes optimized by the CMA-ES algorithm. As a reference we also include the result of JCNN, a joint CNN method.

Figure 10 .

 10 Figure 10. Demosaicing and denoising results on an image from the Imax dataset with σ = 20. We compare the two schemes of DN &DM , cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of DM &1.5DN , MLRI+CBM3D and RCNN+CBM3D, and the MLRI+CBM3D schemes optimized by the CMA-ES algorithm. As a reference we also include the result of JCNN, a joint CNN method.

Figure 11 .

 11 Figure 11. Demosaicing and denoising results on an image from the Imax dataset with σ = 60. We compare the two schemes of DN &DM , cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of DM &1.5DN , MLRI+CBM3D and RCNN+CBM3D, and the MLRI+CBM3D schemes optimized by the CMA-ES algorithm. As a reference we also include the result of JCNN, a joint CNN method.

Figure 12 .

 12 Figure 12. Demosaicing and denoising results on an image from the SIDD dataset. We compare the two schemes of DN &DM , cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of DM &1.5DN , MLRI+CBM3D and RCNN+CBM3D. As a reference we also include the result of JCNN, a joint CNN method.

Figure 13 .

 13 Figure 13. The flowchart of raw image denoising under DM &1.5DN scheme. The dashed VST/IVST blocks are active in just one of the pipeline variants.

Table 1 .

 1 Advantages and drawbacks of the three types of pipelines.

		DN &DM	DM &DN	Joint DM DN
	Advantages	The noise is maintained AWGN	Richer details	Better imaging quality
	Drawbacks	Detail loss and checkerboard artifacts	Spatial and chromaticity-related structural noise	High computational complexity and generalization concerns

Table 2 .

 2 The optimization result of CMA-ES for the pipeline DN 1 &DM &DN 2 (see Eq. (3)), where σ, σ 1 , σ 2 ∈ [0, 255] and α, β ∈ [0, 1]. In this experiment DM is always MLRI and DN is CBM3D or cfaBM3D depending on the input data.

	σ Method	α	β	σ 1	σ 2	CPSNR CPSNR Imax Kodak
		DN &DM	1.00 0.00 5.00	0	34.20	35.08
	5	DM &DN DM &1.5DN 0.00 1.00 0.00 1.00	0 0	5.00 7.50	34.18 34.64	35.03 35.77
		CMA-ES	0.02 0.90	0	7.83	34.66	35.78
		DN &DM	1.00 0.00 10.00	0	31.68	32.15
	10	DM &DN DM &1.5DN 0.00 1.00 0.00 1.00	0 0	10.00 15.00	31.55 32.35	31.62 32.99
		CMA-ES	0.51 0.92 6.81 12.98 32.43	33.02
		DN &DM	1.00 0.00 20.00	0	28.48	28.91
	20	DM &DN DM &1.5DN 0.00 1.00 0.00 1.00	0 0	20.00 30.00	28.07 29.30	27.75 29.85
		CMA-ES	0.52 0.95 10.58 30.63 29.36	29.91
		DN &DM	1.00 0.00 40.00	0	24.90	25.84
	40	DM &DN DM &1.5DN 0.00 1.00 0.00 1.00	0 0	40.00 60.00	24.16 25.46	24.05 26.53
		CMA-ES	0.82 0.98 23.46 41.79 25.74	26.72
		DN &DM	1.00 0.00 50.00	0	23.62	24.83
	50	DM &DN DM &1.5DN 0.00 1.00 0.00 1.00	0 0	50.00 75.00	22.87 24.01	23.00 25.33
		CMA-ES	0.72 1.00 30.55 49.75 24.36	25.61
		DN &DM	1.00 0.00 60.00	0	22.49	23.90
	60	DM &DN DM &1.5DN 0.00 1.00 0.00 1.00	0 0	60.00 90.00	21.83 22.76	22.24 24.26
		CMA-ES	0.90 0.99 34.50 54.42 23.16	24.60

Table 3 .

 3 RMSE between ground truth and demosaicked image for different demosaicking algorithms in presence of noise of standard deviation σ.

	σ	HA	GBTF	RI	MLRI RCNN
	1	5.04	5.10	4.17	4.06	3.21
	3	5.70	5.79	4.97	4.88	4.17
	5	6.78	6.87	6.12	6.10	5.59
	10 10.18 10.27	9.53	9.74	9.65
	15 13.93 14.01 13.15 13.64	13.87
	20 17.75 17.83 16.77 17.56	18.04
	30 25.36 25.42 23.94 25.30	26.21
	40 32.67 32.76 30.77 32.64	33.98
	50 39.58 39.71 37.25 39.55	41.21
	60 46.14 46.35 43.43 46.11	47.95

Table 7 .

 7 The results of different combinations of denoising and demosaicing methods for the Imax image dataset. The best result for each row is red, the second best result is brown, and the third best result is blue.

			DN &DM	DM &1.5DN	CMA-ES
	σ cfaBM3D+ cfaBM3D+ Park+ PCA+ PCA+ RCNN+ MLRI+ cfaBM3D+ JCNN
		MLRI	RCNN RCNN DLMM RCNN CBM3D CBM3D MLRI+
							CBM3D
	5	34.20	35.21	32.86 32.69 34.87 35.44	34.64	34.66	33.48
	10	31.68	32.26	30.06 30.73 31.89 32.77	32.35	32.43	33.09
	20	28.48	28.73	26.86 27.57 27.99 29.54	29.30	29.36	29.79
	40	24.90	24.92	23.86 23.50 23.57 25.69	25.46	25.74	-
	50	23.62	23.59	22.67 22.08 22.10 24.27	24.01	24.36	-
	60	22.49	22.43	21.75 20.89 20.89 23.02	22.76	23.16	-
	Av	27.56	27.86	26.34 26.24 26.89 28.46	28.09	28.29	-

Table 8 .

 8 The results of different combinations of denoising and demosaicing methods for the Kodak image dataset. The best result for each row is red, the second best result is brown, and the third best result is blue.

			DN &DM	DM &1.5DN	CMA-ES
	σ cfaBM3D+ cfaBM3D+ Park+ PCA+ PCA+ RCNN+ MLRI+ cfaBM3D+ JCNN
		MLRI	RCNN RCNN DLMM RCNN CBM3D CBM3D MLRI+
							CBM3D
	5	35.08	36.10	34.87 34.99 35.42 36.58	35.77	35.78	34.13
	10	32.15	32.56	30.85 31.83 32.01 33.36	32.99	33.02	33.27
	20	28.91	29.03	27.42 28.11 28.14 30.12	29.85	29.91	29.95
	40	25.84	25.85	24.88 24.15 24.08 26.82	26.53	26.72	-
	50	24.83	24.83	23.91 22.85 22.77 25.67	25.33	25.61	-
	60	23.90	23.89	23.19 21.77 21.70 24.62	24.26	24.60	-
	Av	28.45	28.71	27.52 27.28 27.35 29.53	29.12	29.27	-

Table 9 .

 9 Average CPSNR results on the SIDD dataset. Note that for each camera, images with different noise levels are being considered. The noise range is σ ∈ [3.28, 38.12]. The proposed DM &1.5DN schemes outperforms the DN &DM ones. The best result is in red, the second best one is in brown. The noise level was estimated by using the method[START_REF] Chen | An efficient statistical method for image noise level estimation[END_REF] and provided to the denoising algorithms and JCNN. Since the sRGB images used in this experiment are already tone-mapped we assumed that the resulting noise is approximately homoscedastic. This allowed us to estimate a single noise level per image instead of a noise curve. Thus, a different noise level was computed for each image in the SIDD sRGB image dataset. The noise estimated for all

	Camera	σ range	JCNN cfaBM3D+ cfaBM3D+ MLRI+ RCNN+
				MLRI	RCNN	CBM3D CBM3D
	IP7	[5.29, 10.65] 36.79	37.30	37.43	37.72	38.37
	S6	[3.71, 38.12] 32.89	33.15	33.31	33.96	33.97
	GP	[3.28, 35.90] 36.42	36.78	37.15	37.52	37.58
	N6	[4.03, 31.15] 33.38	33.96	34.16	34.36	34.21
	G4	[4.66, 13.85] 37.03	37.00	37.20	37.94	37.97
	Av.	[3.28, 38.12] 35.41	35.80	36.00	36.41	36.63
	truth images were acquired by five different mobile phone models. We consid-
	ered the five most effective demosaicing and denoising schemes among those con-
	sidered above, namely cfaBM3D+MLRI, cfaBM3D+RCNN, MLRI+1.5CBM3D,
	RCNN+1.5CBM3D and JCNN.

Table 10 .

 10 Validation of the DM &1.5DN scheme on the SIDD dataset. Note that for each camera, images with different noise levels are being considered. The noise range is σ ∈ [0.48, 22.59] without VST and σ ∈ [0.38, 13.00] with VST. The best result is in red, the second best one is in brown.

			cfaBM3D JCNN	HA+	RCNN+ RCNN+ MLRI+
					CBM3D FFDNet CBM3D CBM3D
	Raw	VST non-VST	49.03 48.53	46.05 45.51	49.18 49.02	48.51 48.55	49.30 49.22	50.55 50.45

Table 11 .

 11 Comparison results of the DM &1.5DN scheme on the SIDD and DND benchmarks (results as reported on the corresponding websites). * indicates the use of the variance stabilizing transform (VST). The best result is in red, and the second best one is in brown.

	Raw	TNRD MLP EPLL WNNM BM3D RCNN+ MLRI+ CycleISP CBM3D CBM3D
	SIDD	42.77 43.17 40.73	44.85	45.52	48.36	49.43	47.98
	SIDD*	-	-	-	-	-	48.56	49.48	-
	DND	44.97 42.70 46.31	46.30	46.64	47.16	47.63	49.13
	DND*	45.70 45.71 46.86	47.05	47.15	47.26	47.76	-

  benchmarks. As with the previous results, the noise ranges of the raw images in the SIDD and DND benchmarks are respectively σ ∈ [0.57, 21.39] and σ ∈ [0.59, 14.97], and after VST the noise ranges are σ ∈ [0.46, 12.79] and σ ∈ [0.44, 9.17], which still satisfy the best

	36.90dB	38.20dB	38.11dB
	TNRD [10]	EPLL [80]	WNNM [27]
	37.84dB	38.44dB	40.07dB
	BM3D [16]	RCNN+1.5CBM3D	MLRI+1.5CBM3D
	36.91dB	36.77dB	38.00dB
	TNRD [10] (VST)	EPLL [80] (VST)	WNNM [27] (VST)
	37.53dB	38.53dB	40.16dB
	BM3D [16] (VST)	RCNN+1.5CBM3D (VST) MLRI+1.5CBM3D (VST)

Table 12 .

 12 Time consumption. The average running time (CPU) of the three strategies in processing 10 images on a PC with an Intel Core i7-9750H 2.60GHz CPU and 16GB memory. Note that we do not use the deep learning methods and only compared the traditional methods. on SIDD, this is likely due to the domain difference between DND and SIDD (as SIDD has darker images). Therefore, this deep learning based approach has several caveats: first MLRI and CBM3D offer guarantees of domain independence and were not trained on the specific image pipeline associated with DND. Second, a difference of 1.5 dB is anyway visually invisible for such high PSNRs as those involved in the table (see Figure

		DN &DM			DM &1.5DN		CMA-ES
	cfaBM3D+ cfaBM3D+ cfaBM3D+	HA+	RI+	MLRI+ cfaBM3D+
	HA	RI	MLRI	CBM3D CBM3D CBM3D	MLRI+
							CBM3D
	7.41 s	7.64 s	7.85 s	16.16 s	16.66 s	16.72 s	23.93 s

Here the CFA image is divided into two half-size RGB images then the noise is removed by CBM3D (see Figure

2).

http://www.cs.yorku.ca/ ~kamel/sidd/benchmark.php

https://noise.visinf.tu-darmstadt.de/benchmark/#results_raw

Acknowledgment. This work was supported by National Natural Science Foundation of China (No. 12061052), Natural Science Fund of Inner Mongolia Autonomous Region (No. 2020MS01002), Young Talents of Science and Technology in Universities of Inner Mongolia Autonomous Region (No. NJYT22090), Innovative Research Team in Universities of Inner Mongolia Autonomous Region (No. NMGIRT2207), Prof. Guoqing Chen's "111 project" of higher education talent training in Inner Mongolia Autonomous Region, Inner Mongolia University Postgraduate Research and Innovation Programmes (No. 11200-5223737), the network information center of Inner Mongolia University, Office of Naval research grant N00014-17-1-2552, DGA Astrid project n • ANR-17-ASTR-0013-01. Y. Guo and Q. Jin are very grateful to Professor Guoqing Chen for helpful comments and suggestions. The authors are also grateful to the reviewers for their valuable comments and remarks.