Axel Clouet

J Ér Ôme Vaillant

David Alleysson

Physical noise propagation in color image construction: a geometrical interpretation

To avoid false colors, classical color sensors cut infrared wavelengths for which silicon is sensitive (with the use of an infrared cutoff filter called IR-cut). However, in low light situation, noise can alter images. To increase the amount of photons received by the sensor, in other words, the sensor's sensitivity, it has been proposed to remove the IR-cut for low light applications.

In this paper, we analyze if this methodology is beneficial from a signal to noise ratio point of view when the wanted result is a color image. For this aim we recall the formalism behind physical raw image acquisition and color reconstruction.

A comparative study is carried out between one classical color sensor and one specific color sensor designed for low light conditions. Simulated results have been computed for both sensors under same exposure settings and show that raw signal to noise ratio is better for the low light sensor. However, its reconstructed color image appears more noisy.

Our formalism illustrates geometrically the reasons of this degradation in the case of the low light sensor. It is due on one hand to the higher correlation between spectral channels and on the other hand to the near infrared part of the signal in the raw data which is not intrinsically useful for color.

Introduction

The most of image sensors are made in CMOS technology containing silicon photodiodes that convert light into electronic signal. They are sensitive to radiations with wavelengths in the visible domain up to near-infrared (NIR) domain (400nm to 1000nm) which corresponds to the absorption range of silicon. To adapt these sensors for color sensing, color filters (RGB) are placed in front of photodiodes to select respectively red, green and blue wavelengths. Unfortunately, common color filters are made of resins which are transparent to NIR requiring to add an infrared cutoff filter (IR-cut) in the sensor's structure.

Photons from a scene are converted in electrons which are stored in an electronic capacity before being converted in readable voltage up to a saturation value. Photon shot noise and readout noise of the sensor are the main sources of uncertainties when we can neglect the dark current. Signal to noise ratio (SNR) is an important criterion when evaluating performances of a sensor. The signal value depends on intrinsic physical parameters of the sensor which can depend on wavelengths λ :

• spectral quantum efficiency of photodiodes (QE ph (λ ))

• transmittance spectra of optical elements including color filters (T f (λ )) • Area of pixels (a 2 pix )

It also depends on other parameters that can be tuned by the user of the camera:

• Integration time or exposure time (t i )

• Aperture of the optics in front of the sensor ( f # )

Signal to noise ratio becomes a critical parameter in low light conditions where integration time and aperture tuning might not be sufficient to increase signal values without altering the image with motion blur. To collect more signal for identical exposure setting (t i and f # ), the sensitivity must be increased. This can be done using larger pixels (spatial widening) but it decreases resolution of the sensor. Another solution is to remove the IR-cut (spectral widening) to take benefit from NIR sensitivity of the silicon[1]- [START_REF] Park | Low light imaging system with expanding spectrum band for digital camera[END_REF]. Color information is then reconstructed computationally from raw data polluted by NIR using a dedicated color conversion matrix (CCM). An example of a sensor specialized for low light contains RGB and W channels (W is a "white" pixel, with no color filter) and no IR-cut, we call it RGBW n (Figure 1). In this paper we compare performances in term of SNR of a classical RGB sensor (with IR-cut) and a RGBW n sensor designed for color sensing under low light conditions [START_REF]Onyx 1.3M -EV76C664 -CMOS Image Sensor -Teledyne e2v[END_REF]. We first recall formalism about signal acquisition, noise nature and color conversion. Next we carry out a quantitative study for both sensors with same physical parameters (except color or infrared filters). An algebraic interpretation of noise propagation helps us to understand why reconstructed color image is dramatically more noisy than raw image for the RGBW n sensor.

Model of signal acquisition and color image formation

Color image formation can be separated in several key steps. Raw signals are acquired all over the color filter array covering the sensor implying physical parameters mentioned in the introduction [START_REF] Joyce | Image Systems Simulation[END_REF]. Then raw data are spatially interpolated through the demosaicing algorithm which is not under scope here [START_REF] Alleysson | Linear demosaicing inspired by the human visual system[END_REF] [START_REF] Amba | N-LMMSE demosaicing for spectral filter arrays[END_REF]. Finally, the interpolated raw image is converted in a color image using a color conversion matrix (CCM) which maps the intrinsic input spectral space given by the camera to an output color space. We choose CIE-XYZ 1931 [START_REF] Bianco | Color correction pipeline optimization for digital cameras[END_REF]- [START_REF] Vaillant | Color correction matrix for sparse RGB-W image sensor without IR cutoff filter[END_REF] as reference for output space because one can then easily convert XYZ coordinates in any display space such as sRGB. The image formation flow is schemed Figure 2.

Raw data acquisition Analytical formalism:

A scene contains for each location different objects with an associated reflectance spectrum ρ(λ ) which is illuminated by a light source I(λ ) that we decompose such as I(λ ) = N lux .i(λ ) where i(λ ) is the spectral distribution of I(λ ) normalized to 1 lux and N lux a factor representing the illumination level in lux. For each pixel the number of collected electrons can be written as follows [START_REF] Joyce | Image Systems Simulation[END_REF]:

M e-= N lux .t i .a 2 pix 4. f 2 # . ∞ -∞ i(λ ).ρ(λ ).QE ph (λ ).T f (λ ).dλ (1)
This value is altered by an uncertainty mainly due to the photonic shot noise (variance σ 2 ph = M e-) dominant at high light levels, and readout noise (constant variance σ 2 rn ) of the sensor, dominant at low light level.

Algebraic formalism:

The goal of this section is to rewrite equation 1 in term of matrix and vectors. We start defining a function containing the input light information: L(λ ) = N lux .i(λ ).ρ(λ ) and another containing effective quantum efficiencies of spectral channels QE(λ ) = QE ph (λ ).T f (λ ). Then we define a small wavelength sampling step δ λ and discretise curves in a n number of wavelengths (for sensitivity of silicon, it is not zero on the range 400 -1050 nm, δ λ = (1050 -400)/(n -1) in nm). L and QE are now column vectors containing the n samples (in R n ) of L(λ ) and QE(λ ) [START_REF] Cohen | Visual color and color mixture: the fundamental color space[END_REF]. Equation 1 can be rewritten in an algebraic form as:

M e-≈ M e-= k × QE T .L , with k = t i .a 2 pix .δ λ 4. f 2 # . (2)
So the signal M e-is proportional to the orthogonal projection of L on QE (scalar product). In this equation the "." operator is a classical matrix product. Now we assume that for any sensor, its spectral sensitivities generate a free family of vectors which are generally not orthogonal. For p different spectral channels, the family k × (QE 1 , ..., QE p ) is a basis of a p dimensions vector space in R n (p n) so, for writing simplification, we note measurement vectors f i = k × QE i with i = {1, ..., p}. Similarly, we write M e-,i the values associated to the signals in the f i basis. These values are the orthogonal projections of light vector L in the sensor's spectral space. A geometric illustration of the measurement process for a simple bi-dimensional (p = 2) case is proposed in Figure 3.

Color conversion

A color conversion matrix is a linear operation that transforms color information extracted from raw data acquired by a sensor to the output color space. It is mathematically built as a composition of operators dedicated respectively to spectral information extraction and then color data computation.

Spectral content of the raw measurement

From light spectrum L ∈ R n , the spectral content L of the measurement is the projection of L in the sensor's space. To get the explicit form of L in R n , we can write it as a sum vector of f i which are known elements in R n . However, as the sensor's basis is usually not orthogonal, L = ∑ p i=1 M e-,i .f i . These measured coordinates are called "covariant". We must find the associated so called "contravariant" coordinates M i [START_REF] Dullemond | Introduction to tensor calculus[END_REF] to write L = ∑ p i=1 M i .f i (see Figure 4). Let's define a n × p measurement operator F with F = [f 1 ...f p ] such as [M e-,1 ... M e-,p ] T = F T .L [START_REF] Marimont | Linear models of surface and illuminant spectra[END_REF]. Contravariant coordinates can be found by applying the inverse of the correlation matrix of the basis vectors on the covariant coordinates [START_REF] Cohen | Visual color and color mixture: the fundamental color space[END_REF] [START_REF] Dullemond | Introduction to tensor calculus[END_REF]:

[M 1 ... M p ] T = (F T .F) -1 .[M e-,1 ... M e-,p ] T (3)
Explicitly the total operation of spectral information extraction can be written using F, [START_REF] Drew | Multispectral processing without spectra[END_REF][15]:

L = F.(F T .F) -1 .[M e-,1 ... M e-,p ] T (4)

Color component

The chosen color space is CIE-XYZ 1931 which is defined by the color matching functions (CMF) displayed Figure 5. CMF can be sampled over the same wavelengths as previous data and put in a 3 × n operator H = [x y z] [START_REF] Jaaskelainen | Vector-subspace model for color representation[END_REF]. CIE XYZ coordinates of the measured input light vector L can be determined by applying H on L. The same way as the measurement done by the sensor, this corresponds to an orthogonal projection of L on the "color axes" (scalar products).

[ X Ŷ Ẑ] T = H T . L (5)
Figure 5: Color matching functions. The projection of the light vector measured by a sensor on a color axis is illustrated Figure 6. Finally a 3 × p matrix operator which applies on covariant coordinates (raw measurements) can be derived from equations 5 and 4, we call this operator color conversion matrix (CCM kernel ) which is explicitly written in equation 6 [START_REF] Finlayson | Metamer constrained color correction[END_REF]:

CCM kernel = H T .F.(F T .F) -1 (6)
This operator is called "kernel" because it does not take in consideration neither display space (sRGB, Adobe RGB,...) which requires other next linear combination, nor normalization which is necessary when we consider acquisition and display digital range. Nevertheless, as the main objective of this paper is to analyse noise propagation through color reconstruction process, we may skip these linear and normalization steps.

Noise propagation: computational data and interpretation

To illustrate the impact of noise, we chose two distinct sensors : a standard RGB sensor (with IR-cut) and a four channel sensor (RGBW n without IR-cut). The only changing parameter between these sensors are the spectral sensitivity shapes due to color filters and IR-cut. is an RGBW n sensor without IR-cut designed for low light conditions [START_REF]Onyx 1.3M -EV76C664 -CMOS Image Sensor -Teledyne e2v[END_REF]. We will compare performances of such a sensor with a classical RGB sensor (actually based on the same spectral responses but attenuated with an infrared cutoff filter and without W channel). Spectral sensitivities of those sensors are shown in Figure 7.

We arbitrary consider a relatively low light level at N lux = 10 lux. For a white reflectance (flat spectrum) seen under the equal-energy illuminant CIE-E (extended to 1050 nm), 99% of the saturation limit of the W pixel (which is the most sensitive channel) is reached to an integration time about t i = 12ms. This value will be kept to perform a fair comparison between RGBW n and RGB sensors. However note that for this sensor, the red channel is the more sensitive and reaches its saturation at t i = 59ms. For a more complete comparative study, we may have chosen to also compare the case where RGB sensor does not have the IR-cut or a RGBW one equipped with the IR-cut. However, these two additional sensors both have poor performance making them out of consideration of this paper. When IR-cut is removed from RGB sensor, spectral properties are not well designed to reconstruct spectra correctly. As a result, false colors from NIR components imply a slight drop in term of color performance for the CIE-XYZ 1931 output space. Alternatively, if we add a white channel without removing the IR-cut, the W channel is unfortunately very close to a linear combination of R,G and B channels. So the correlation matrix depicted equation 3 is singular and hardly invertible leading to an high amplification of noise in spectral reconstruction. For these reasons, we only keep the classical RGB with IR-cut and the RGBW n without IR-cut that have similar performances in term of color rendering.

Comparative raw noise

Raw measurements include signal and noise. Signal is calculated as in equation 2, and two sources of noise are considered.

Photon shot noise (σ 2 ph ) and readout noise (σ 2 rn ). For one single pixel, signal to noise ratio can be computed using equation 7. When signal is equal to 10000 e-, SNR = 39.95dB, corresponding to shot noise limited SNR.

SNR = 20 × log S var(S) = 20 × log   M e- σ 2 ph + σ 2 rn   (7)
Raw signal to noise ratio is evaluated at the output of the sensor without color conversion. As shown in Table 2 the wider the spectral bands, the higher the raw SNR.

Noisy color image reconstruction

To compare the relative noise propagation between the two sensors, RGB and RGBW n , we simulate acquisitions using a hyperspectral reflectance image with the two considered sensors. The hyperspectral image is picked up in our own database, containing reflectance data of a scene over the range of wavelengths from 400 nm to 1050 nm at a 20 nm step. Integration time and illuminant (extended CIE-E) are the same and color images are extracted using two color conversion matrices corresponding to each sensor. These images are shown Figure 8. In both cases the calculated CCM is satisfactory as color reconstruction seems correct but the image is more noisy in the case of RGBW n sensor than for the RGB classical sensor.

To quantify signal to noise ratio of the color converted image, we use a Monte-Carlo approach. Many noisy raw acquisitions are simulated in same conditions for a perfectly flat reflectance spectrum under the considered illuminant.

Typically data are extracted statistically over 5000 simulated acquisitions. For each sensor, the CCM is applied and we focus on the Ŷ component of the result [START_REF] Mornet | Evaluation of color error and noise on simulated images[END_REF], [START_REF] Mornet | Toward a Quantitative Visual Noise Evaluation of Sensors and Image Processing Pipes[END_REF] (which is called the "luminance" component of the color image). Signal value is the mean value of the projected Ŷ points, noise is their standard deviation. Quantitative results are summarized in Table 3.

Table 3 In opposite to the SNR of raw signal, SNR on color extracted information is far more noisy in the case of the RGBW n low light sensor than in RGB classical one.

Geometrical interpretation

The loss of signal to noise ratio when converting raw images into color images can be related to two main reasons that can be graphically illustrated. First reason is correlation between channels which is higher in the case of RGBW n sensor than in classical RGB (see Figure 7). The second is linked to the NIR part of the acquisition (case of RGBW n sensor) which is not a useful part of the acquired raw information for color reconstruction.

Effect of correlations between channels

From a geometrical point of view, the more the channels are correlated, the smaller are the angles between measurement vectors. As each measurement point is altered by an uncertainty on measurement axes, the zone of uncertainty given by two correlated channels is larger than the zone of the uncertainty given by two less correlated channels. The impact of uncertainty and correlation is illustrated on Figure 9 for a two-dimensional case.

Figure 9: Uncertainty of spectrum locus (grey area) due to noise in the case of weakly correlated channels (top) and highly correlated channels (bottom).

In Figure 9, weak correlation corresponds to RGB sensor whereas strong correlation represents RGBW n sensor. As illustrated, the spectrum locus uncertainty is larger in the case of strongly correlated channels (grey area) as in RGBW n sensor.

As an indicator of this spectral uncertainty, we can quantify SNR on the contravariant coordinates corresponding to each sensor. As previously, we use a Monte-Carlo approach to compute these results: many noisy measurements are simulated (equation 1), then, corresponding contravariant coordinates of each measurement is computed (equation 3). For each channel, we extract mean measurement and its corresponding standard deviation. Graphically this is equivalent to project the grey area of Figure 9 alongside sensor axes (see Figure 4). Table 4 shows that SNR in contravariant coordinates are dramatically decreased in the case of RGBW n sensor compared to RGB sensor when they are considered under same conditions.

Effect of joint visible and NIR acquisition

As explained earlier and shown in Figure 6, the Ŷ value is the orthogonal projection of L on the y axis which is only defined on the visible domain (Figure 5). This uncertainty on the Ŷ value is the orthogonal projection of the L uncertainty locus on the y axis. The shape of the locus and the angle between y and measurement axes can lead to a loss of SNR on Ŷ especially when these angles are wide. In our examples, RGBW n sensor acquires both visible and NIR wavelengths, raw channels are less correlated to color axes than in the case of the RGB sensors which acquires only visible wavelengths. Then angles between measurement vectors and color axes is wider for RGBW n sensor. In Figure 10, we illustrate a case where channels are highly correlated between them, and more or less correlated to the y axis.

In Figure 10, the case of RGBW n sensor is more accurately represented by the top situation where color signal is diluted in a raw signal containing visible and NIR at the same time.

As an illustration of this loss, we consider only the two green channels of both sensors as mono-channel sensors (denoted G with the IR-cut, G n without the IR-cut). As shown previously, measurement points (corresponding to a perfect white reflectance under extended CIE-E illuminant) are projected on the y axis. We still use the Monte-Carlo approach to compute signal to noise ratios. Results are compiled in Table 5.

Conclusion

Color image sensors have been optimized for low light conditions by widening spectral sensitivities to acquire visible and NIR information at the same time. Using this method, raw SNR are higher but the corresponding color reconstructed image appears more noisy.

To understand how noise propagates, the algebraic model of color reconstruction from raw data have been depicted to geometrically illustrate the raw measurement, the reconstruction of the spectral information and color information. By representing noise on raw acquisition, it is possible to understand why reconstructed image is more noisy in the case of the RGBW n sensor compared to RGB sensor.

The loss is due on one hand to higher correlations between spectral channels in the case of the RGBW n sensor which widens the uncertainty locus of the measured spectral data. On the other hand it is due to the NIR part of the signal which is not beneficial for color reconstruction because of the lack of correlation between XYZ color axes and sensor's basis. Considering the different simulations which has been computed, the effect of the correlation between spectral channels seems to be the most critical when going from the RGB to the RGBW n sensor.

Finally, widening spectral bands to NIR implies several compromises between having high SNR on raw images or on color images. Understanding the reasons of noise propagation can lead to several prospects. From a physical point of view, mathematical and geometrical models can guide physical optimizations of sensors according to the wanted applications. From signal point of view, the understanding of noise propagation can be used to design more adapted algebraic operation for color correction. In addition, post-processing algorithms can be designed to use the denoised raw image to help improving the corresponding noisy color image. By the way, even if this study was restricted to color sensing, all geometrical considerations can be used for multispectral imaging.

Figure 1 :

 1 Figure 1: Example of classical RGB color sensors with IR-cut (left), example of RGBW n low light sensor (right).

Figure 2 :

 2 Figure 2: Image processing flow.

Figure 3 :

 3 Figure 3: Illustration of the measurement of L in the sensor's space for a two-dimensional case.

Figure 4 :

 4 Figure 4: Illustration of the position of the contravariant coordinates in a two-dimensional case.

Figure 6 :

 6 Figure 6: Illustration of the projection of the measured vector on a color axis (y) denoted Ŷ for a two-dimensional case.

Figure 7 :

 7 Figure 7: Spectral sensitivities of the RGBW n low light sensor (top) without infrared cutoff filter and a standard RGB sensor equipped with the infrared cutoff filter (bottom).

Figure 8 :

 8 Figure 8: Color image at saturation limit for RGB sensor (top) and RGBW n (bottom).

Figure 10 :

 10 Figure 10: Representation of Ŷ signal and uncertainty in the case where sensor channels are correlated between each other, and channels are weakly correlated to y axis (top), or highly correlated (bottom)

Table 1

 1

		summarizes common physical
	parameters.
	Table 1: Physical parameters of the sensor
	1. Pixel size (a pix )	10 µm
	2. Optics aperture ( f # )	1.8
	3. Saturation (S max )	10000 e-
	4. Readout noise (σ rn )	10 e-

Spectral sensitivities of the investigated sensor are the same as those of the Onyx sensor manufactured by Teledyne E2V which

Table 2 :

 2 Raw signal to noise ratio for both sensors.As expected, SNR are significantly better on each raw channel for the RGBW n sensor compared to RGB sensor since spectral sensitivities have been widened to NIR domain.

	Channels RGBW n	RGB (with	RGB (with
		(no IR-cut)	IR-cut)	IR-cut)
	@ t i	12 ms	12 ms	59 ms
	R	38.1 dB	32.6 dB	39.8 dB
	G	36.8 dB	32.8 dB	39.8 dB
	B	36.4 dB	32.6 dB	39.9 dB
	W	39.9 dB

  : SNR on Ŷ channel for RGB and RGBW n sensors.

	Sensors	RGBW n	RGB (with	RGB (with
		(no IR-cut)	IR-cut)	IR-cut)
	@ t i	12 ms	12 ms	59 ms
	SNR Y	26.1 dB	32.4 dB	39.4 dB

Table 4 :

 4 Signal to noise ratio of contravariant coodinates for RGBW n and RGB sensors

	Channels RGBW n	RGB (with	RGB (with
		(no IR-cut)	IR-cut)	IR-cut)
	@ t i	12 ms	12 ms	59 ms
	M R	9.8 dB	26.6 dB	34.1 dB
	M G	8.8 dB	19.8 dB	27.4 dB
	M B	16.4 dB	25.0 dB	32.4 dB
	M W	12.8 dB

Table 5 :

 5 Signal to noise ratio of the Y color information contained in G and G n channels, θ y/g denotes the geometric angle between the y color axis and the raw G axes of the two sensors

	Channels G n (no IR-cut)	G (with IR-cut)
	@ t i	12 ms	59 ms
	θ y/g	50.7 •	18.1 •
	SNR raw	39.95 dB	39.95 dB
	SNR y	36.9 dB	39.7 dB

Author Biography

Axel Clouet received his Master degree in semi-conductors physics and photonics from Grenoble Institute of Technology -Phelma (France) in 2017. He is currently pursuing his PhD at the department of optics and photonics of the CEA-LETI of Grenoble. His interest is now focused on signal and image processing for color and multispectral imaging.