Haris Ahmad Khan
email: haris-ahmad.khan@u-bourgogne.fr

Jean-Baptiste Thomas

Jon Yngve Hardeberg

Olivier Laligant

Spectral adaptation transform for multispectral constancy

Keywords: Multispectral imaging, reflectance reconstruction, multispectral constancy, illuminant estimation, spectral adaptation transform

The spectral reflectance of an object surface provides valuable information of its characteristics. Reflectance reconstruction from multispectral image data is typically based on certain assumptions. One of these common assumptions is that the same illumination is used for system calibration and image acquisition.

We propose the concept of multispectral constancy which transforms the captured sensor data into an illuminant-independent representation, analogously to the concept of computational color constancy. We propose to transform the multispectral image data to a canonical representation through spectral adaptation transform. The performance of such a transform is tested on measured reflectance spectra and hyperspectral reflectance images. We also investigate the robustness of the transform to the inaccuracy of illuminant estimation in natural scenes. Results of reflectance reconstruction show that the proposed spectral adaptation transform is efficient and is robust to error in illuminant estimation.

Introduction

The formation of an image depends on the spectral reflectance of the surface being viewed, the spectral power distribution and intensity of the illumination, and the spectral sensitivities of the sensors. In the human visual system, the three cone types act as sensors. In case of a camera with 3 channels, the RGB filters together with the imaging sensor, play a similar role. The human visual system has the natural ability to perceive constant color of surfaces despite the change in spectral composition of the illuminant, and this ability to discard illumination effects is called Color Constancy [START_REF] Foster | Color constancy[END_REF]. Creation of such models for illuminant invariant representation of scenes in computer vision is called computational color constancy [START_REF] Gijsenij | Computational color constancy: Survey and experiments[END_REF]. An illuminant invariant representation is important for computer vision applications including object recognition, tracking and image classification [START_REF] Barnard | A comparison of computational color constancy algorithms -Part I: Methodology and experiments with synthesized data[END_REF]. There are two major techniques for achieving computational color constancy. One method is to compute illuminant invariant features, and a second method is to estimate the illuminant and later apply a correction [START_REF] Hordley | Scene illuminant estimation: Past, present, and future[END_REF]. In this paper, we use the former method of illuminant estimation.

The advancement in sensor technology has developed the use of multispectral imaging for indoor and close range imaging. The ability of multispectral imaging in acquisition of more spectral information is useful for object and material classification and identification by means of spectral reconstruction [START_REF] Imai | Spectral estimation using trichromatic digital cameras[END_REF][START_REF] Hardeberg | Acquisition and Reproduction of Color Images: Colorimetric and Multispectral Approaches[END_REF][START_REF] Connah | Recovering spectral information using digital camera systems[END_REF][START_REF] Valero | Recovering spectral data from natural scenes with an RGB digital camera and colored filters[END_REF][START_REF] Hardeberg | Multispectral colour imaging: Time to move out of the lab?[END_REF] of surfaces in a scene. The need for spectral reconstruction of surfaces was recognized in 1980s [START_REF] Maloney | Evaluation of linear models of surface spectral reflectance with small numbers of parameters[END_REF][START_REF] Maloney | Color constancy: a method for recovering surface spectral reflectance[END_REF][START_REF] Hall | Illumination and color in computer generated imagery[END_REF][START_REF] Parkkinen | Characteristic spectra of Munsell colors[END_REF][START_REF] Jaaskelainen | Vector-subspace model for color representation[END_REF]. Since then, many methods are developed to provide spectral reconstruction from the camera data. Most of these methods rely on the use of training data to learn the mapping between the camera data and the desired spectra. This process is called calibration of the system and is performed through a training set of measured reflectances and radiance data with a given illuminant. To maintain a reasonable accuracy, the same illuminant is required for scene acquisition. This limitation of having the same illuminant for calibration and image acquisition is a major shortcoming for generic use of multispectral imaging [START_REF] Shrestha | Spectrogenic imaging: a novel approach to multispectral imaging in an uncontrolled environment[END_REF].

To overcome this limitation, we propose to transform the acquired multispectral data under any unknown illumination into its canonical representation. This transformation requires the estimation of scene illuminant. For multispectral images, estimation of illuminant in the sensor domain by using image statistics based methods is proposed by Thomas [START_REF] Thomas | Illuminant estimation from uncalibrated multispectral im-ages[END_REF] and Khan et al. [START_REF] Khan | Illuminant estimation in multispectral imaging[END_REF]. In their work, the illuminant estimation methods for color images are extended from three channels to K channels. The estimated illuminant is used for the diagonal correction [START_REF] Finlayson | Color constancy: generalized diagonal transforms suffice[END_REF] to transform the input sensor data into a canonical representation. We call this consistent representation of multispectral data as multispectral constancy. A preliminary proposal of this concept is provided in [START_REF] Khan | Multispectral constancy based on spectral adaptation transform[END_REF].

To achieve multispectral constancy, we use a diagonal transform and also introduce a spectral adaptation transform (SAT). The concept of SAT being proposed is closely related to the data based sensor sharpening transform by Finlayson et al. [START_REF] Finlayson | Spectral sharpening: sensor transformations for improved color constancy[END_REF]. The main difference between these strategies is that instead of finding a sharpening transform and then diagonalizing it, we want to optimize the result after applying the diagonal transform, where the elements of diagonal transform are obtained from illuminant estimation in the sensor domain. The proposed idea of multispectral constancy and optimization of diagonal transform is tested with a simulated multispectral camera on the reflectance dataset and hyperspectral images of real scenes from the Foster dataset [START_REF] Foster | Frequency of metamerism in natural scenes[END_REF]. The advantage of having a canonical representation is that the spectral reconstruction system can be calibrated with a canonical illuminant and hence, the condition of having same illuminants for training under which the scene is acquired, is no longer needed. Results show a promising aspect of the use of multispectral imaging for outdoor scene acquisition under uncontrolled illumination condition.

This paper is organized as follows. In Section 2, we define multispectral constancy after formalizing a model for the multispectral image acquisition system. System calibration and spectral reconstruction are defined as linear problem in the section. Section 3 defines our experimental protocol based on simulations.

Results are analyzed in Section 4 before we conclude.

Multispectral constancy

Definition

In a simplified noiseless imaging model, formation of an image depends on the spectral sensitivity of imaging sensor c(λ), spectral reflectance of the surface r(λ) and the spectral power distribution of illuminant e(λ). This formation for the visible wavelength spectrum ω is defined as

f = ω r(λ)e(λ)c(λ)dλ (1)
In practice, we can formulate an extended and discrete version of Eq. 1 as;

F = REC (2)
Considering the spectral sampling (N ) of 10nm within the wavelength range of 400nm to 700nm and K number of spectral channels, F is S × K matrix (S is the number of spectral samples), R is S × N matrix of surface reflectance, E is the diagonal matrix (N × N ) of the scene illuminant and C is N × K matrix, consisting of spectral sensitivities of the channels.

Here we consider two cases of image acquisition with the same imaging system. One image is acquired with a canonical illumination E c and another image with an unknown illuminant E ill . We present both cases in parallel in Eq. 3.

F ill = RE ill C ; F c = RE c C (3)
To perform the spectral reflectance estimation R from the imaged data in both of the above mentioned cases, a generalized inverse, denoted by + , can be applied as;

R = F ill C + E + ill ; R = F c C + E + c (4)
However, C is not necessarily a square matrix and therefore, it is not trivial to compute the inverse. Computational procedures are applied to achieve the task of spectral reconstruction. There are several works in literature where the spectral reconstruction is performed by linear transform through a calibration matrix. Linear methods are popular for learning the mapping between camera data and desired output (spectral reflectance). The idea behind using linear methods is that when the reflectance spectra is continuous and band-limited [START_REF] Maloney | Evaluation of linear models of surface spectral reflectance with small numbers of parameters[END_REF][START_REF] Parkkinen | Characteristic spectra of Munsell colors[END_REF], the statistical analysis of measured reflectances of standard color samples are enough to calibrate the spectral reconstruction system [START_REF] Jaaskelainen | Vector-subspace model for color representation[END_REF][START_REF] Zhao | Image-based spectral reflectance reconstruction using the matrix R method[END_REF].

By using the calibration matrix W, the equations for spectral reflectance re-

construction become R = F ill W ill and R = F c W c for both cases, respectively.
This calibration is specific for a given illumination E c . The scene illuminant plays a direct role in the spectral reconstruction [START_REF] Connah | Recovering spectral information using digital camera systems[END_REF][START_REF] Eslahi | Recovery of spectral data using weighted canonical correlation regression[END_REF], because of its effect in the camera image formation, as in Eq. 1.

With F c , the spectral reconstruction system can work efficiently, since the calibration is already performed with the same canonical illuminant, but with F ill , the calibration matrix needs to be recomputed with E ill . Measurement of scene illuminant and calibration matrix for each change in imaging environment is a difficult task and is not a practical solution for a situation where the illumination is not constant all the time. To avoid this problem, we propose to transform the acquired multispectral data F ill into a canonical representation F c . Thus, W c can be used for the spectral reconstruction from the multispectral data, being taken under any illumination. We denote such a transform as M c,ill , which maps the camera data F ill , taken under unknown illuminant E ill , into its canonical representation F c .

F c = M c,ill F ill . ( 5
)
Once such a transform is available, then the problem of spectral reconstruction is limited to finding the transform M c,ill . The spectral reconstruction in this case is mathematically represented as;

R = W c M c,ill F ill (6)
We call this concept of illuminant invariant representation of multispectral data as multispectral constancy. By achieving multispectral constancy, the requirement of having the same illumination for calibration and scene is no longer required.

In this work, we propose the use of illuminant estimation for achieving multispectral constancy. In [START_REF] Khan | Multispectral constancy based on spectral adaptation transform[END_REF], a diagonal transform is used as M c,ill and the preliminary results for spectral reconstruction from the sensor data were provided. Such a transform requires the estimation of scene illuminant in the sensor domain and the sensor data is corrected from effects of illumination through a diagonal transform (D). The diagonal matrix D c,ill is a K × K matrix and the components of this matrix are the sensor responses to the illuminations E c and E ill . As an alternate explanation, sensor response to the illuminants can be understood as the sensor response to a perfect white diffuser under a certain illuminant, and denoted by w c and w ill for the two illuminants. The matrix D is defined as;

D c,ill = diag(w c /w ill ) (7)
With the use of diagonal transform as in Eq. 7, the problem of color constancy (for 3-channel images) and multispectral constancy (for multispectral data) is reduced to the estimation of K parameters of the diagonal transform. D is applied on each channel independently and is used in many color constancy algorithms [START_REF] Finlayson | Shades of gray and colour constancy[END_REF][START_REF] Van De Weijer | Edge-based color constancy[END_REF].

With a perfect white diffuser, Eq. 5 holds and the input illuminant is transformed into the desired canonical illuminant with a diagonal transform. However, when the surface reflectance is not constant across the wavelength spectrum, then this transformation generates errors. The diagonal transform also works well when the bandwidth of spectral sensitives of filters are within 100 to 150 nanometers [START_REF] Worthey | Heuristic analysis of Von Kries color constancy[END_REF][START_REF] Forsyth | A novel algorithm for color constancy[END_REF]. However, in case of large band filters, the diagonal transform may not be sufficient. Therefore, we investigate the performance of diagonal transform and propose to minimize the error during diagonal transformation through a spectral adaptation transform A SAT . The idea of SAT is to incorporate the sensors response for efficient transformation, along with the diagonal transform. The concept of spectral adaptation transform is analogous to chromatic adaptation transform [START_REF] Hunt | Chromatic adaptation transforms[END_REF], while here we consider multispectral data, rather than the three-channel color images.

Related state of the art

To improve the performance of M for color images, Finlayson et al. [START_REF] Finlayson | Spectral sharpening: sensor transformations for improved color constancy[END_REF] proposed sensor sharpening which aims at finding a linear combination of the spectral sensitives of a camera, with respect to which a diagonal transform for illuminant transform works the best. The idea in this technique is to transform the sensor responses so that they appear to be sharper than the original ones. Finlayson et al. [START_REF] Finlayson | Spectral sharpening: sensor transformations for improved color constancy[END_REF] proposed three methods for finding the sharpening transform T. The first method is called sensor based sharpening, and finds T by optimizing the intuitive notion of sharpness for each filter individually. This optimization requires the knowledge of sharpening interval in the wavelength spectrum and the Lagrange multiplier [START_REF] Finlayson | Spectral sharpening: sensor transformations for improved color constancy[END_REF]. The second method is called perfect sharpening, in which it is assumed that the surface reflectances can be fitted in a three dimensional linear model while the illuminants can be fitted in a two dimensional linear model. The third method is data based sharpening, in which RGBs are generated from a set of known spectral reflectances, using a specific camera and two illuminations, one as canonical illumination E c while the other as test illumination E ill . The sharpening transform T is added on both sides of Eq. 5 to reduce the error in mapping. It is expressed as;

TF c = MTF ill . ( 8
)
According to Eq. 8, M can be optimal in the least-square sense if it can be defined by the Moore-Penrose inverse ( + );

M = TF c [TF ill ] + (9)
Since T is still unknown in Eq. 9, the unknown terms are shifted to the left hand side;

T -1 MT = F c [F ill ] + (10)
The sharpening transform T is found through eigenvector decomposition of the expression on left side of Eq. 10. In their work, the expression [T] -1 MT is diagonalized so that M is replaced by a diagonal matrix D. Finlayson et al.

[20] found similar T for sensor based sharpening and data based sharpening.

The sharpening transform can sometimes have negative values and there are some investigations to reduce this problem [START_REF] Vazquez-Corral | Spectral sharpening of color sensors: Diagonal color constancy and beyond[END_REF]. Data based sharpening was further improved by Barnard et al. [START_REF] Barnard | Sensor sharpening for computational color constancy[END_REF] by using an average of measured illuminants, and introduced a parameter for prioritizing the positivity. Drew and

Finlayson [START_REF] Drew | Spectral sharpening with positivity[END_REF] proposed data driven positivity by adding constraints to ensure that all the values in sharpening transform are positive. Chong et al. [START_REF] Chong | The von Kries hypothesis and a basis for color constancy[END_REF] proposed the measurement tensor technique for finding T. Spherical sampling [START_REF] Finlayson | Spectral sharpening by spherical sampling[END_REF] is also introduced as a technique for spectral sharpening where the discretely sampled points on a sphere are found and related to the original sensors.

Sharpening through filter chart calibration was proposed by Abdellatif [START_REF] Abdellatif | Physics-based spectral sharpening through filter-chart calibration[END_REF]. An overview of spectral sharpening methods is provided by Corral and Bertalmío [START_REF] Vazquez-Corral | Spectral sharpening of color sensors: Diagonal color constancy and beyond[END_REF].

In case of multispectral imaging, the concept of spectral sharpening is not straightforward due to the higher dimensionality of sensors. In [START_REF] Finlayson | Spectral sharpening by spherical sampling[END_REF], spectral sharpening for six multispectral sensors is discussed. Their method is computationally expensive since there is need to generate a sphere of the sensor dimension which is not trivial. For a dimension above 3, the spherical sampling [START_REF] Finlayson | Spectral sharpening by spherical sampling[END_REF] has to be extended to hyper-sphere computation. Due to these complications, we do not use the sensor based sharpening method. We do not use the data based sharpening method either, because we are not interested in diagonalizing the result of Eq. 10. Instead, we are using D as the diagonal matrix containing the illuminant in the sensor domain, as in Eq. 7, and to improve the efficiency of such a transform in the same way as in spectral sharpening. It can be argued that instead of improving the diagonal transform, why not find any other linear transform which can serve the same purpose more efficiently? The reason for preferring the diagonal transform (and its improvement) in our work is the fact that knowledge of scene illuminant is of major importance in computer vision applications. Therefore, either it can be measured for a specific scene or can be estimated, and this information can be used by our proposed method. Another reason is that most of the color constancy algorithms are defined in terms of a diagonal transform, where the elements of diagonal transform are found through illuminant estimation in the sensor domain.

For achieving illuminant invariant representation in multispectral imaging, Abrardo et al. [START_REF] Abrardo | Color constancy from multispectral images[END_REF] proposed the concept of color constancy for multispectral imaging by linearly transforming the sensor response under an unknown illuminant. The transformation matrix M is determined through the least-square solution as;

M = F c F + ill = F c F t ill (F ill F t ill ) -1 (11)
The transformation M is applied to the acquired multispectral data and then spectral reflectance is estimated. It is interesting to note that the right side of Eq. 11 resembles with the concept of data based sharpening (Eq. 10). This method works well when the canonical representation (F c ) for the same scene is available. In [START_REF] Abrardo | Color constancy from multispectral images[END_REF], authors claim that 97% accuracy in spectral reconstruction is achieved by their method. The problem with this technique is that in the absence of F c , it is not possible to use Eq. 11. In fact, our aim is to transform the input multispectral data into its canonical representation, while in Eq. 11, the availability of canonical representation of same scene is assumed, which is not feasible for every image being captured.

Proposal for computation of M

It can be observed that Eq. 10 and 11 are originally formulated from Eq.

5 and the goal is to find the transform M. In spectral sharpening, M is found by eigenvector decomposition of Eq. 10 and diagonalizing it, while the product of F c and F ill is used directly in Eq. 11. We propose decomposing M into two elements; D c,ill as in Eq. 7 and A SAT . In this proposed method, the diagonal transform performs the transformation of multispectral data taken under unknown illuminant E ill , into its canonical representation under the illuminant E c . The role of A SAT is to incorporate the spectral response of imaging sensors and improve the transformation of camera data into its canonical representation. The optimal A SAT should minimize the error for all reflectances i and illuminants j.

F i c = A SAT D c,j F i j ( 12
)
To find the best A SAT , Eq. 12 is written as the explicit minimization of an error function in Eq. 13;

min

A SAT = j |F i c -A SAT D c,j f i j | (13)
For a given sensor configuration, a generic A SAT is created by minimizing the error for all the reflectances and illuminations in the training dataset.

The advantage of our proposed technique is that only the knowledge of sensor sensitives of camera is required to compute A SAT . This specific A SAT for a camera can be used to transform the multispectral data captured under any illuminant, into its canonical representation through the diagonal correction for illumination. The only parameter left in the proposed system is the performance of illuminant estimation algorithm, since the elements of diagonal transform are obtained through the illuminant estimation.

The multispectral data being acquired under any illumination is transformed into a canonical representation after estimating the scene illuminant, and then the pre-calibrated W c is used. The advantage of our proposed idea of multispectral constancy is that it is no more necessary to acquire the multispectral data in a controlled environment.

Once the transform M c,ill = A SAT D c,ill is defined, spectral reconstruction can be mathematically represented as;

R = W c A SAT D c,ill F ill ( 14
)
Preliminary results by using the diagonal transform D c,ill are provided in [START_REF] Khan | Multispectral constancy based on spectral adaptation transform[END_REF].

In the current work, we are using D c,ill along with A SAT . The experimental protocol for testing the idea of multispectral constancy through SAT is provided in the following section.

Experiments

To implement and validate the idea of multispectral constancy through SAT, we use measured reflectance data of 1995 surfaces from the SFU reflectance dataset [START_REF] Barnard | A data set for color research[END_REF]. These surfaces include the 1269 Munsell chips, 24 patches of the X-rite ColorChecker, 170 natural objects [START_REF] Vrhel | Measurement and analysis of object reflectance spectra[END_REF], 120 Dupont paint chips [START_REF] Vrhel | Measurement and analysis of object reflectance spectra[END_REF], 350 surfaces from the Krinov dataset and 57 surfaces measured by Barnard et al. [START_REF] Barnard | A data set for color research[END_REF]. This dataset is used as the training data for calibration of W c with D65

as the canonical illuminant. The reflectance data is in the wavelength range of 400 to 700 nm with 10 nm sampling.

For testing the proposed idea of multispectral constancy, we use 24 patches from the X-rite ColorChecker and 1296 Munsell chips as the test data and acquire them using the simulated multispectral sensors. Spectral reconstruction is done from the acquired multispectral data as in Eq. 14. To validate the usefulness of proposed idea for natural outdoor images, we use the reflectance data from Foster dataset of hyperspectral images [START_REF] Foster | Frequency of metamerism in natural scenes[END_REF]. These hyperspectral images are within wavelength range of 400 nm to 720 nm, but we use these images within wavelength range of 400 nm to 700 nm, since the training dataset [START_REF] Barnard | A data set for color research[END_REF] is within this range. In the following sections, the details of experimental setup are provided.

Sensor

Diagonal transform works well when the sensor sensitives are narrow-band (ideally Dirac delta functions). Following this, it seems as the use of narrow band filters is the optimal choice to be used in imaging systems. The term F c F t ill in Eq. 11 resembles the spectral sharpening, introduced by Finlayson et al. [START_REF] Finlayson | Spectral sharpening: sensor transformations for improved color constancy[END_REF]. They proposed a linear combination to transform the original sensor responses into narrower bands. However, from a practical point of view, one reason for not employing such narrow band systems is that the acquisition time, complexity and cost of such system is high as compared to wide band systems [START_REF] Shrestha | Evaluation and comparison of multispectral imaging systems[END_REF]. Other reasons are the fact that the narrow band systems are not an optimal choice for illuminant estimation [START_REF] Thomas | Illuminant estimation from uncalibrated multispectral im-ages[END_REF][START_REF] Khan | Illuminant estimation in multispectral imaging[END_REF] and demosaicing (when multispectral filter array is used [START_REF] Lapray | Multispectral filter arrays: Recent advances and practical implementation[END_REF]). Wang et al. [START_REF] Wang | Multispectral imaging: narrow or wide band filters?[END_REF] studied the influence of increase in number of bands and found that increasing the number of spectral bands cause reduction in performance of spectral reconstruction. Also, the efficiency of illuminant estimation algorithms decreases when the number of spectral filters is increased [START_REF] Khan | Illuminant estimation in multispectral imaging[END_REF]. In this work, we are limiting the experiments to linear systems and our aim is to investigate the concept of multispectral constancy for a generalized multispectral imaging system. We use 5, 8 and 12 spectral bands for testing the proposed framework of spectral constancy. In the results of illuminant estimation in multispectral images by Khan et al. [START_REF] Khan | Illuminant estimation in multispectral imaging[END_REF], the accuracy of illuminant estimation is reduced when the number of spectral bands is increased beyond 8. On the other hand, having more than 8 bands still increase the spectral estimation. For this reason, we limit the number of bands within this range for our experiments.

In our experiments, we use a Gaussian model of sensor sensitivities. Such a model has been extensively used in the literature to simulate sensors or to approximate Fabry-Pérot filter transmittance [START_REF] Lapray | Energy balance in spectral filter array camera design[END_REF]. This configuration is called the equi-Gaussian [START_REF] Thomas | Illuminant estimation from uncalibrated multispectral im-ages[END_REF]. This sensor configuration is tested for illuminant estimation [START_REF] Khan | Illuminant estimation in multispectral imaging[END_REF] and spectral estimation [START_REF] Khan | Multispectral constancy based on spectral adaptation transform[END_REF]. The Full Width at Half Maximum (FWHM)

of the sensor sensitivities decrease with increase in number of bands and the overlap between adjacent bands remain approximately the same. By increasing the number of bands (K) in this configuration, we are gradually shifting from typical multispectral sensors towards hyperspectral sensors.

For testing the proposed method of SAT for spectral reconstruction, we also use measured sensitivities of a real implementation of spectral filter array (SFA) camera [START_REF] Thomas | Spectral characterization of a prototype SFA camera for joint visible and NIR acquisition[END_REF]. There are eight filters in this SFA camera. The first six filters are used in our experiments as the available spectral reflectance data for training (SFU dataset) is within the wavelength range of 400 nm to 700 nm. Figure 1 shows the spectral sensitivities of each filter being used in the experiments.

A SAT computation

For a given sensor configuration, A SAT is computed by using the set of 102 illuminants and surface reflectance of 1995 surfaces from the SFU data [START_REF] Barnard | A data set for color research[END_REF].

Radiance data is generated using an illuminant j (from the set of 102 illuminants) and is acquired as multispectral data. For each illuminant, a diagonal transform is created by using Eq. 7 and the multispectral data under that illuminant is transformed by using Eq. 5. The desired output after applying M c,ill is the multispectral data under the canonical illuminant (D65). As discussed in Section 2.3, we propose to decompose M c,ill into diagonal transform D c,ill , and A SAT . This A SAT is unique for a given sensor configuration and is computed by using Eq. 13. Once A SAT is computed, Eq. 12 is used to transform F ill into F c . A SAT for each sensor configuration being tested in this work is provided in the supplementary data file (Supplementary_Data.xlsx).

Illuminant estimation

For testing and validating the proposed concept of spectral adaptation transform, we keep D c,ill fixed for the spectral reflectance data. For obtaining the elements of D c,ill , we use the test and canonical illuminants in the sensor domain. Initially we want to validate the efficiency of A SAT while testing the reflectance dataset, therefore we assume that the illuminant is estimated in the sensor domain without error.

In practical cases, the information about scene illumination may not be available all the time. This is where the illumination estimation algorithms come into play. By using an illuminant estimation algorithm, the elements of diagonal transform are obtained and then the pre-calculated SAT is used.

For testing the proposed concept for natural scenes, we perform illuminant estimation in the multispectral data of natural scenes and use the Max-Spectral Algorithm, which is the extension of Max-RGB algorithm [START_REF] Land | Lightness and Retinex Theory[END_REF]. The extension of this algorithm from color to spectral is proposed and analyzed in detail in [START_REF] Thomas | Illuminant estimation from uncalibrated multispectral im-ages[END_REF][START_REF] Khan | Illuminant estimation in multispectral imaging[END_REF]. The estimated values of illuminant, which are defined in the sensor domain, are used in the diagonal transform D c,ill . For each image, D c,ill is estimated individually and this estimation may consist of error as well. We report the error in illuminant estimation in form of angular error (∆A).

Spectral reflectance reconstruction

As explained in Section 2, a calibration matrix W is required for the spectral reconstruction from camera data. It is obtained by using measured reflectance spectra R t and the camera sensor sensitivities (M). For reducing the error between original spectra R and the estimated spectra R, a covariance matrix of a set of measured reflectance samples can be used. Those measured reflectance samples provide the a-priori statistical information about the surfaces in a scene [START_REF] Conde | Spectral reflectance estimation of ancient Mexican codices, multispectral images approach[END_REF]. If the a-priori information is well chosen, error in the spectral reconstruction can be minimal.

There are several methods being proposed for the spectral reconstruction in literature [START_REF] Connah | Comparison of linear spectral reconstruction methods for multispectral imaging[END_REF]. We use a linear method for clarity, namely the Wiener estimation [START_REF] Shen | Reflectance reconstruction for multispectral imaging by adaptive Wiener estimation[END_REF] because of its robustness to noise. It is defined as

W = R t R T t (CE) T ((CE)R t R T t (CE) T + G) -1 . ( 15
)
Here, R t R T t and G are the autocorrelation matrices of the training spectra and additive noise, respectively. G is in the form of a diagonal matrix consisting of the variance of noise σ 2 . Training for obtaining the matrix W c is performed with CIE illuminant D65 as the canonical illuminant E c . The obtained calibration matrix is used for spectral reconstruction in Eq. 14.

Evaluation

For testing the proposed idea of multispectral constancy, radiance data is simulated from the measured test spectra with CIE illuminants A, D50, D55, D75, F5, F7 and F12. We also use the LED (Philips SlimStyle: 2700K) as an illuminant in the experiments.

The proposed idea is also tested on hyperspectral images of scenes consisting of vegetation and urban areas. We create radiance data from these images using illuminants A, D50, D55 and D75. The same procedure of reflectance reconstruction for each pixel of simulated multispectral image is performed and evaluated.

To measure the performance of the spectral reconstruction, we compare the reconstruction r for each patch of the reconstructed reflectance with the corresponding measured reflectance r, through root mean square error (RMSE)

as

RMSE = 1 N N j=1 (r j -rj ) 2 (16)
We also use goodness of fit coefficient (GFC) [START_REF] Hernández-Andrés | Colorimetric and spectroradiometric characteristics of narrow-field-of-view clear skylight in Granada, Spain[END_REF] for evaluation of spectral reconstruction results. For each reconstructed reflectance r, GFC is calculated as

GFC = r T r (r T r)(r T r) . ( 17
)
In case of the multispectral images from Foster dataset, RMSE is calculated for each pixel of the estimated reflectance, with the original reflectance in the hyperspectral image. We also compute the colorimetric error by a linear mapping of the reflectance data into its corresponding CIEXYZ. The CIEXYZ is converted into CIELab by taking white point of D65. Error between CIELab from the original reflectance and reconstructed reflectance is calculated in terms of CIEDE2000 [START_REF] Cie | Improvement to industrial colour-difference evaluation[END_REF]. For each evaluation metric, we include three methods. First method is by doing nothing to the input multispectral data and using Eq. 15 for spectral reconstruction. We call this method as do nothing. In second method, simple diagonal transform is applied to the input data before using it for spectral reconstruction, while in the third experiment, the input data is first transformed by using the proposed SAT. Results are discussed in Section 4.

Results

The results of spectral reconstruction of Munsell reflectance data, by using eight different illuminations and four different sensor configurations, are shown in form of graphs in Fig. 2, consisting of mean RMSE, mean GFC and mean CIEDE2000. For each test illuminant, we compare the spectral reconstruction results after applying diagonal transform to the input multispectral data, SAT and do nothing. The RMSE results show that for illuminant A, the diagonal transform reduces the error as comparing to do nothing, but for illuminants D50, D55 and D75, applying only a simple diagonal transform significantly increases the error in spectral reconstruction as compared to when the SAT is applied. There is slight increase in RMSE with the use of diagonal transform for illuminants F5, F7 and F12, while there is no change in error when LED is used as illuminant source. These results show that in terms of RMSE, applying diagonal transform to input multispectral data increases the error in comparison to do nothing. This trend is consistent among the 5, 8 and 12 channel cameras being used. Results obtained from simulated 6 channel SFA camera also show similar results. When our proposed SAT is used along with diagonal transform on the input multispectral data, then RMSE is reduced significantly.

By increasing the number of channels, there is further reduction in RMSE for all illuminants except F12, where RMSE is minimum when 8 channels are used.

Overall, the significant reduction in RMSE shows the efficiency of our proposed SAT for spectral reconstruction.

When the spectral reconstruction is evaluated in terms of GFC, diagonal transform and SAT perform closely. Do nothing performs lower for illuminants A, F5, F12 and LED while the performance difference is less significant for other test images. There is no change in performance by increasing the number of channels, except for illuminant F12, where the performance of do nothing goes down by increasing the number of channels. The increase in error for do nothing is because the bands become more sensitive to illumination changes when they get narrower.

Colorimetric evaluation of the spectral reconstruction reveals the interesting fact that there is no difference in do nothing and applying the diagonal transform. Error in terms of CIEDE2000 for both diagonal transform and do nothing, is increased slightly when the number of channels are increased from 5 to 12.

However, applying SAT significantly reduces the colorimetric error, as can be seen in Fig. 2. The performance of SAT behaves opposite to other two methods and error is slightly reduced with increase in number of channels.

For testing the proposed idea of multispectral constancy on images contain- ing real scenes, we use the Foster hyperspectral reflectance data [START_REF] Foster | Frequency of metamerism in natural scenes[END_REF]. We use four different sensor configurations as simulation of multispectral camera for acquiring the data. Sensor configuration for these cameras is shown in Fig. 1.

Before acquiring the multispectral data, each hyperspectral reflectance image is converted into radiance image by applying an illuminant. We use illuminants A, D50, D55 and D75.

Illuminant estimation in each multispectral image is performed by using the Max-spectral algorithm [START_REF] Khan | Illuminant estimation in multispectral imaging[END_REF]. Diagonal transform is applied to each multispectral image with the estimated illuminant. That estimated illuminant contain error which is evaluated in terms of angular error (∆A). Fig. 3 show the performance of spectral reconstruction in terms of mean RMSE, mean GFC and mean CIEDE2000 of the eight images being tested. Detailed results along with ∆A are provided in the supplementary data file.

With the Foster dataset, the error in spectral reconstruction is larger than the results of Munsell dataset. The main reason is the error in illuminant estimation. As the elements of diagonal transform are an estimation of the illuminant, therefore the error in illuminant estimation is intensified in the spectral reconstruction as well. Fig. 3 show RMSE results for 5, 8, 12, and 6 (SFA) channels, respectively. Do nothing and simple diagonal transform produces almost similar results except for multispectral data acquired with 8 channels. In that case, simple diagonal transform performs better as compared to do nothing. RMSE is reduced significantly when SAT is used, which shows that our proposed method performs effectively in reducing RMSE.

In terms of GFC, do nothing performs better than simple diagonal transform, except for illuminant A, where the do nothing performs significantly lower. Same trend can be seen across all four sensor configurations being tested. SAT is able to perform slightly better than diagonal transform and do nothing. Evaluation in terms of CIEDE2000 shows that do nothing and simple diagonal transform performs same except for 5 channels, where error is increased when diagonal transform is used. By using the proposed SAT, there is decrease in CIEDE2000 which shows that the proposed idea is valid with images containing real scenes.

Although the colorimetric error is still large, but it should be kept in mind that inaccuracy in illuminant estimation also play its role in the overall error in spectral reconstruction. The average RMSE with 5 and 12 channels are almost equal and the same result in obtained with SFA, while RMSE is comparatively larger when 8 channels are used. In terms of colorimetry, the error is reduced gradually by increasing the number of channels. When simple diagonal transform is used then CIEDE2000 remains almost equal for 8 and 12 channels while the error is reduced for all sensor configurations when SAT is used. Therefore, by increasing the number of filters, there is slight improvement in the spectral reconstruction results. However, the performance may become more sensitive to the imaging noise. Those results are based on simulations and are still to be validated for the experimentally captured multispectral data. Detailed results from the experiments, including the angular error in illuminant estimation for each image, mean and maximum errors of RMSE, GFC and CIEDE2000, are provided in the supplementary data file.

In most of the illuminant estimation algorithms proposed in literature, the efficiency of algorithm is evaluated in the terms of angular error ∆A and a diagonal transform is applied to the input images. In most of the cases, the transformed images appear to be taken under a canonical (usually white) illuminant and it is assumed that the effect of scene illuminant is removed through the diagonal transform. Results from our experiments show that a simple diagonal transform is not sufficient. For illustration, we show an example of radiance image from Foster dataset and the corresponding transformations (Fig. 8. In this example, the color rendering of radiance spectral image with test illumination and reconstructed spectral images (after applying diagonal transform and SAT), are shown. D55 is used for creating the radiance scene and multispectral image is acquired with 12 equi-Gaussian channels Illuminant estimation gives ∆A of 0.1013. As can be seen in Fig. 8, the effect due to D55 is removed after the transformations and the output images appear to be almost the same visually but the RMSE, GFC and CIEDE2000 evaluation shows that there is difference among these 2 images in terms of spectra and colorimetry. This suggests that the effect of such transformations has to be verified in computer vision applications.

To visualize the reconstructed spectra, eight examples are provided in Fig. The reconstruction results using simple diagonal transform and SAT match closely, while the do nothing results are significantly lower, as they are not close to the measured spectra in terms of shape and scale. This observation shows that with illuminant estimation and then applying the transformation into canonical representation has the clear advantage. Upon a careful comparison, it can be seen that by applying SAT, the scale of reconstructed spectra is brought closer to the original spectra. This explains the reason that why our proposed SAT is able to reduce the error in spectral reconstruction. The visualization of spectra shows that with the use of SAT, there is no significant difference in the shape of reconstructed spectra. The main difference is in scale of intensity, which explains that why GFC remains same while RMSE and CIEDE2000 metrics provide different results. One of the drawbacks of using linear method for spectral reconstruction is its limitation in reconstruction of spectra containing spikes.

Linear spectral reconstruction assumes smooth spectra and provides a smoothed approximation to the input spiky spectra. This effect can be seen in Fig. 4b applications, such as object detection and material identification.

Conclusion and perspectives

In this work, we propose the concept of multispectral constancy, which de- The colorimetric difference also needs to be investigated so that an acceptable level of accuracy for color based systems can be achieved. Finally, it has yet to be investigated that what level of accuracy is required in the spectral reconstruction so that the estimation can be used for spectral information based applications. Such an application can be object detection and identification on the basis of its spectral properties. Hence. our proposed idea of multispectral constancy is opening research questions which needs to be investigated, in order to enable the generic use of multispectral imaging for real-world applications.

Improvement in results can be done by carefully selecting the training data which can represent the objects in the scene. In this way, the SAT and spectral reconstruction system can be calibrated for a specific scenario. We did not work on improvement in selection of training data in this work as our main aim is to study the effectiveness of proposed idea of multispectral constancy and SAT under general conditions.

Simulation results show that for unimodal sensitivities and linear sensor, the proposed SAT is an adequate transform and permits to achieve efficient reflectance reconstruction. However, when the SAT is evaluated based on an estimate of illumination, error in illuminant estimates makes the performance to decrease. It is still to be investigated that what accuracy is required for illuminant estimation to make this concept beneficial. Further work shall investigate these directions and define the limits of using this approach.

Figure 1 :

 1 Figure 1: Spectral sensitivities of the filters being used

Figure 2 :

 2 Figure 2: Evaluation of spectral reconstruction results of Munsell spectral reflectance. The column on left show results obtained from 5 channel camera, 2 nd column consist of results obtained from 8 channel camera, column 3 show results obtained from 12 channel multispectral data while the last column show results obtained from 6 channel SFA camera. These results show that by using the proposed A SAT , error in spectral reconstruction

Figure 3 :

 3 Figure 3: Evaluation results from the Foster dataset. Among the results, column 1 show results obtained with 5 channel multispectral camera, column 2 consist of results with 8 channel multispectral camera, column 3 show results obtained from 12 channel multispectral data while the last column show results obtained from 6 channel SFA camera.

4, 5 , 6 & 7 .

 567 In each Fig. two sample spectra for one image of Foster dataset are shown. These samples are selected on the basis of best and worst GFC values. Each Fig. provides the comparison between the measured spectral reflectance and the spectral reconstruction results with do nothing, simple diagonal transform and SAT. For each figure, the measured values of ∆A, GFC and RMSE are provided in the captions.

Figure 7 :Figure 8 :

 78 Figure 7: Spectral reconstruction of 2 sample spectra from Foster dataset (image 8). Illuminant D75 is used to create radiance data, simulated multispectral camera with 6 SFA channels is used and illuminant estimation error ∆A is 0.0884

  fines the illuminant invariant representation of multispectral images. Multispectral constancy is achieved through a spectral adaptation transform (SAT), which transforms the data representation from an unknown illuminant, to a canonical one. This SAT is obtained by training the system. Advantage of the proposed technique is that the only information required is the sensor sensitivities of imaging system.With the idea of multispectral constancy being developed, what needs further attention is the establishment of efficient illuminant estimation methods for multispectral imaging. The proposed framework for spectral reconstruction is sensitive to accuracy in illuminant estimation. This dependency is seen when spectral reconstruction is performed on the images from Foster dataset. The performance of SAT is affected with error in illuminant estimation and this dependency may be addressed through development of efficient illuminant estimation algorithms. Once such an efficient illuminant estimation algorithm in the sensor domain is available, our proposed SAT is able to show efficient results as demonstrated in the simulations.

0.2 0.3 0.4 0.5 0.6 0.7 Measured Spectrum Do nothing Simple Diagonal Transform Spectral Adaptation Transform

  Spectral reconstruction of 2 sample spectra from Foster dataset (image 4). Illuminant D50 is used to create radiance data, simulated multispectral camera with 8 equi-Gaussian filters is used and illuminant estimation error ∆A is 0.0373

		0.3
					Measured Spectrum
		0.25			Do nothing Simple Diagonal Transform
					Spectral Adaptation Transform
		0.2
	Intensity	0.6 0.8 0.05 0.1 0.15	Do nothing Spectral Adaptation Transform Simple Diagonal Transform Measured Spectrum					4 6 8	# 10 -3			Spectral Adaptation Transform Simple Diagonal Transform Do nothing Measured Spectrum
	Intensity	0.2 0.4 400 0	450	500	Wavelength 550	600	650	700	Intensity	0 2
		0								-2
		400 -0.2	450	500	550	600	650	700		400 -4	450	500	550	600	650	700
					Wavelength									Wavelength
	(a) GFC with simple diagonal transform:	(b) GFC with simple diagonal transform:
	0.9985, GFC with SAT: 0.9990.			0.3848, GFC with SAT: 0.3956.
	RMSE with simple diagonal transform:	RMSE with simple diagonal transform:
	0.0562, RMSE with SAT: 0.0023			0.0013, RMSE with SAT: 0.0001
	, 5b, 6b & 7b, where the linear reconstruction is unable to address the spikes in the measured spectra. These observations lead to the question that what is the purpose of color and multispectral constancy. If it has to be applied for visual correction only, then a simple diagonal transform may be efficient, as seen in Fig. 8. If the purpose is to reconstruct the spectral reflectance, then a simple diagonal transform is not 400 450 500 550 600 650 700 0 0.1 Wavelength 400 450 500 550 600 650 700 Intensity -0.005 0 0.005 0.01 0.015 0.025 0.03 Measured Spectrum Do nothing Simple Diagonal Transform Figure 5: Wavelength 0.02 Spectral Adaptation Transform
	helpful while GFC metric indicates that there is slight improvement when simple (a) GFC with simple diagonal transform: (b) GFC with simple diagonal transform:
	0.9977, GFC with SAT: 0.9981.			0.7652, GFC with SAT: 0.7666.
	diagonal transform is used. Evaluation in terms of CIEDE2000 indicates that RMSE with simple diagonal transform: RMSE with simple diagonal transform:
	0.0202, RMSE with SAT: 0.0036			0.0036, RMSE with SAT: 0.0013
	the results with do nothing and diagonal transform are the same. However,
	Figure 6: Spectral reconstruction of 2 sample spectra from Foster dataset (image 6). Illumi-
	all the evaluation metrics show that the proposed SAT is able to reduce the nant A is used to create radiance data, simulated multispectral camera with 12 equi-Gaussian
	filters is used and illuminant estimation error ∆A is 0.0545
	error significantly. Having said this, the next goal is to investigate that what is
	the role of improving the currently used evaluation metrics in computer vision