Professor Kacem Chehdi

Ludovic Macaire

Albert Dipanda

Pierre Gouton

Jon Hardeberg

Iii Forewords

Keywords: Multispectral imaging, Spectral lter arrays, imaging pipeline

This manuscript is intended to put my research on spectral lter arrays imaging in perspective. To the reader who wish to learn more about me than about my research, I recommend starting with the Chapters 6 and 7, where an overview on my research and curriculum vitae are presented.

A special thought is going to Pierre-Jean Lapray. We met for an adventure that is not ended yet! I am particularly proud that from a post doc under my responsibility he became Associate Professor and a good friend.

Jon Yngve Hardeberg and all my colleagues at the Colorlab. Jon, thanks for the many opportunities that came up along the way.

Sabine Süsstrunk and her team at EPFL. It has been great working with you.

It has also been challenging to change home every 2 weeks during one year, but I would do it again.

Thanks to Keivan Ansari who came from Iran as a Post Doc. I am sure we are going to have more collaborations in the future.

I want to thank the PhD students I supervised: they had to endure me and I am sometimes very direct. Also the Professors who helped me to supervise them. I learnt a lot about research and sta management, both in the Human and administrative sides, Pierre and Jon, but also Alamin Mansouri, Olivier Laligant and Marius Pedersen. Thanks to the graduate and undergraduate students who interacted with me also. It is always interesting to have to explain what you do and to interact with young newcomers and fresh ideas. You keep me young! Many co-authors are already listed above, but for those who are not: I think of you while I am writing this document. Also, I would like to thank the colleagues who helped me to make this document better with a critical reading of my drafts. I am thinking particularly to Alain Trémeau who provided me with very useful comments on the form. I am thinking also about Steven Le Moan, Pierre-Jean Lapray and Aditya Sole who helped to improve the language. The Le2i had historically developed multispectral imaging systems (e.g. [Mansouri 2005]), but has also hosted strong research with applications to robotics and medical imaging, therefore it has a strong link to computer vision. I joined this laboratory in 2010 as Maître de Conférences (permanent position of Associate Professor), with a background in colour imaging, and the spectral lter arrays technology, which is developed here, is very much a melting pot of those dierent inuences. Locally most of the research was conducted in collaboration with Professor Pierre Gouton, Professor Alamin Mansouri and Professor Olivier Laligant. They have also helped me to supervise my PhD students as Professors.

I used several funding schemes in synergy to support my research. Funding schemes ranged from local small funding dedicated to the development of a specic item, to EU projects based on large consortium, through industrial applied projects. This is summarised in Chapter 7 where the reader will nd a list of the projects and the dierent roles I had within them. Those fundings permitted to hire temporary research stas, and I had co-directed 4 PhDs and had 2 post-docs to work with me, who are also listed in Chapter 7.

The work at Le2i has been strengthen by specic interactions with two noteworthy academic partners, namely the Norwegian Colour and Visual Computing Laboratory (CVCL) at NTNU in Norway, and the Images and Visual Representation Lab Please see next Chapters and in particular Chapter 7 for more details on the context. Table 1.1 is a timeline description of the research presented in this manuscript.

Imaging technology of the visible range is related to several research elds and, in my opinion, is essentially trans-disciplinary. An image is a representation of the world, and so its capture, processing and essence can be considered from dierent perspectives; e.g. Acquisition itself depends on physics and electronics; Image processing is related to signal theory; Image understanding may depend on cognitive psychology; Tools to handle images are usually linked to computer science; Models may be based on applied mathematics, physics or computer graphics and deep learning. Extensions can also be found into design and graphics, and many applications could be considered from a point of view of sociology or at least Humanities in the large sense. Although, the colour and spectral imaging eld is a research niche, it is very rich in diversity of backgrounds and perspectives. Nevertheless, this has an impact on the size of the community and thus on its visibility. I will develop this aspect related to publications later in Section 1.3.

Multispectral imaging

For this work, I dene and limit multispectral imaging as the close-range acquisition of an image of a scene within the visible and potentially near-infrared (NIR) part of the electromagnetic spectrum based on an arbitrary spectral sampling. Dierences with hyperspectral imaging is that I do not assume that the spectrum is sensed by Dirach functions, nor even approximated by very narrow bandwidths. Dierences with remote sensing is that I limit the sensing to the visible and NIR, and I do not assume a complex atmospheric model, neither a more complex acquisition model than what is described below. Colour imaging is a particular case of multispectral imaging, where the spectral sampling aims at the acquisition of primaries that permits the reconstruction of colour information, i.e. RGB, using trichromaticity theory.

In this context, the radiance, s(x, λ), reaching the camera from the object at position x is integrated over a set of spectral sensitivities of the camera c(λ), and gives a set of values f (x), as described in Eq. 1.1. In general, there is a digitization function applied to f (x) so that the sensor delivers a digital value, often with a precision between 8 and 14 bits. x is in the image domain Ω ⊂ R 2 , λ ∈ [380, 1100] nm is in the spectral domain (visible and NIR), where this model is most likely to represent relatively well the physics. If c(λ) is somehow standardised, such as often assumed for colour camera 1 or greylevel camera, then the relation between f (x) and the intensity of the radiance may be straightforward. This is more dicult in the general case due to the diversity of spectral sensitivities.

f (x) = s(x, λ)c(λ)dλ (1.1)
If we consider the hypothesis of diuse materials and light source, then the radiance is the contribution of the global spectral power distribution of the illumination, e(λ), and the spectral reectance, r(x, λ), of the surface, as in Eq. 1.2. This equation is the basis for many spectral reconstruction methods, which aim at the reconstruction of r(x, λ) from f (x). This is an ill-posed problem, but several assumptions, such as smoothness of spectral reectance and sometimes uni-modality of the sensitivities permit to compute a good approximation. This also implies to know e(λ) and c(λ).

f (x) = r(x, λ)e(λ)c(λ)dλ (1.2)
If the diuse material hypothesis does not hold, as in most cases, then one may use a dichromatic reectance model, which considers a specular, σ, contribution from the illumination in addition to the diuse, δ, component, such as in Eq. 1.3. The dichromatic model has been dened by Shafer [Shafer 1985] for colour images and generalised to spectral by [START_REF] Tominaga | Standard surface-reectance model and illuminant estimation[END_REF]]. This equation is the basis for computer vision, which aims at separating object surface properties from illumination and shadows in the image. In this document it may be used in Chapter 4 to estimate the illumination, from highlights for instance.

f (x) = δ(x) r(x, λ)e(λ)c(λ)dλ + σ(x) e(λ)c(λ)dλ (1.3)
Those equations are usually enough to understand most of the literature and simulations about multispectral imaging. In some cases, e(λ) would be e(x, λ) to account for spatial variation of illumination and shadows. I have not addressed specically this aspect yet. We note that there is no uorescence involved in the model, and I do not 1 In color images, the three sensitivities are often dened for colour estimation and can vary while they still stand for Red, Green and Blue. consider it in the following. We also note that there is no sub-scattering involved in the model, and I do not consider it in the following. Those aspects are tremendously important, and I do not discard them lightly in this manuscript. Some applications should consider and incorporate a more complex model.

The spectral nature of illumination and camera sensitivities are very important, and a system calibration is often required to use multispectral imaging, as well as the control of illumination. A vast body of literature addresses those topics, readers may start their review by the recent book from M. Kriss et al. [Kriss 2015], and then relate to the subsequent literature. Note also two of the major handbooks of the color imaging eld [Sharma 2002, Lee 2005], which will provide useful insights.

Because there are several sensitivities in multispectral imaging, the tempo-spatiospectral sampling will need to sacrice one of those dimensions to capture such images.

It is done by either reducing

• spectral resolution by reducing the number of c(λ),

• time resolution, while performing a sequential acquisition that will reduce the number of frames per second and generate potential needs for pixel registration,

• spatial resolution, and generate also spatial mismatch between bands.

I will go back to this in the next Section. It is also to be noted that, not unlike colour imaging 20 years ago, multispectral imaging is becoming increasingly accepted and used by the computer vision and robotic communities, in particular, articles are published around the specic case of RGB-NIR imaging or case study based on existing commercial multispectral sensors.

Scientic publications on multispectral imaging

In support of this, we can observe on Figure 1.1 the elds related to the keyword multispectral imaging in the web of science interface. We can note the diversity of applications and the traditional scientic disciplines that are concerned with the eld. I should mention that many of these publications is more concerned by remote sensing. On Figure 1.2, we can observe the increased number of publications related to this eld, in particular after 2011. This is partly due to the progress of the acquisition technology, which permits to enable this modality of imaging for diverse applications (so the communications are not only based on its development). This is also due to the increasing number of scientic publications worldwide linked to the trend to quantity, to the electronic publishing, but also due to the emergence of new very active countries (e.g. China) in our research elds.

It is interesting to compare the dierent elds that show up when we refer to hyperspectral imaging (Figure 1.4), where remote sensing and spectroscopy are becoming prominent. It is also informative to refer to colour imaging (Figure 1.3), where computer science is then becoming prominent.

Conferences in this eld are traditionally focused on colour imaging (e.g. Colour and Imaging Conference -CIC, Electronic Imaging -EI) or image processing (e.g. ICIP).

Spectral lter arrays

Spectral Filter Arrays (SFA) is one of the several proposals to capture multispectral images. It is essentially a generalisation of the Colour Filter Arrays (CFA), which is very well known for its Bayer instantiation [Bayer 1976]. In this case, we sacrice more of the spatial resolution, traded for time eciency and increased spectral sampling. A graphical example is given in Figure 1.5, where we can observe the principal components of a SFA: the lters, a monochrome sensor and a process to reconstruct the full-resolution image.

Advantages and disadvantages are along those trade-o: We can capture multispectral images at video rates without mechanical or optical registration problems, but we are limited in the number of bands in order to preserve a useful spatial resolution. We are also constrained in sensitivities to ensure that enough photons are integrated by the sensors. The main advantage is the easy encapsulation into a classical imaging pipeline with a rather simple optical set-up and a single solid-state sensor. Direct competitive technology would be light-eld cameras, which would have a simpler spectral ltering process due to the size of lters, but more complex optical set-up and paths [Lam 2015].

Overview on related research on this technology

There is a large history of research to nd the best CFA possible that considers more than three primaries and specic arrangements, e.g. [Hirakawa 2008]. This research targets the best colour image acquisition. To my knowledge, explicit SFA for spectral imaging was proposed in [START_REF] Ramanath | [END_REF][START_REF] Ramanath | [END_REF], then modeled in [Miao 2006b, Miao 2006a] from a binary tree perspective. Many following works have been proposed, but only a small number of practical instances have been implemented.

In terms of practical implementation, typical successful teams include an industrial/governmental and a research center plus potentially a commercialisation module, e.g. Tanaka-Lab and Olympus, IMEC [IMEC 2018] and Ximea 2 , Hirakawa (Univer- sity of Dayton, Ohio) and US Air-force [Hirakawa 2017], we collaborated with Silios 2 https://www.ximea.com/ 1.6. Overview on scientic contributions related to this manuscript 9 [SILIOS-TECHNOLOGIES 2018], but we also observe instrument manufacturers, e.g. Pixelteq [PIXELTEQ 2018]. Although the combinations varies in term of arrangement and engagement, the key point is that the realisation is calling to such a broad expertise that it would be very dicult to nd all the skills and experts in one single institute.

Dierent instances assumed dierent starting hypothesis. Monno et al. [Monno 2015] targeted colorimetric imaging and relighting with their ve bands instantiation of SFA. Hirakawa targeted spectral reconstruction with his Fourier sensor [START_REF] Jia | [END_REF], Hirakawa 2017]. We targeted general computer vision with our visible and NIR sensor [Thomas 2016b]. IMEC developed an approximated hyperspectral imager by using a number of narrow-bands either in the visible or in the NIR [IMEC 2018].

I should mention, in addition, SFA cameras for short-wave infrared (SWIR) imaging [START_REF] Kutteruf | [END_REF], Kanaev 2015].

Kanaev and his team dened also the imaging pipeline including demosaicing and super-resolution for still images and for videos within the SWIR. Sensors based on arrays of polarimeters and processings [Andreou 2002[START_REF] Lemaster | [END_REF]] have been developed too.

Overview on scientic contributions related to this manuscript

My PhD thesis focuses on the colorimetric modelling of displays (visualisation devices) [Thomas 2009a], which is a sub-eld of colour imaging. From there, I linked this research with the colorimetric modelling of scanners (acquisition devices) during my post doc at the C2RMF. When I joined the Le2i, I started to develop a research on cameras, and extended from colour to multispectral imaging. This was made within a tentative to increase the amount and the quality of information in order to provide better visual information, whatever visual would mean here (for a computer vision system or for a human observer). I always kept in mind that for me, the human (as observer or user) should remain at the centre of my research paradigm. With this in mind, I still conducted parallel research in colour imaging and colorimetry. And a very logical forward move was to investigate material appearance by means of imaging technologies. Each of those aspects are developed and referenced in Chapter 6.

If I should summarise and structure our research contributions during that period, I would cluster them into three aspects: Contributions to colour imaging and processing, contributions to prototype and experimental data generation, contributions to computational spectral imaging (demosaicing, white balance, etc.). In fact, exploratory works and experimental data acquisition are a very important part of this research. This also explains the diversity of methods and models that were used: I focused on the type of data, not on a specic model.

The resulting object is a group of techniques and data that permit to propose the use of multispectral imaging in general computer vision tasks. Indeed, I provide prototype instances and experimental data that enable real-time acquisition simulations and applications. I also provide tools to handle those data until they could be understood and handled by the computer vision community, although there is still a strong need for standardization. I provide in parallel a link to the visualisation of those data, although I remain so far in a colorimetric image research paradigm on this aspect. An analysis of the weaknesses and strengths of the dierent research items is provided along the three next Chapters. An analysis of the limits of the overall research and perspectives are discussed in Chapter 5.

Structure of this manuscript

The technical research results are published and accessible to the community. For this manuscript, I selected only three papers with pivotal contributions, which permit to discuss the other papers in perspective of the whole research. This permits to have a compact manuscript, while the other publications are available online for the interested reader. I give directions and overviews on the topics and aim at providing a higher level point of view on the problems. This manuscript is thus a collection of papers. Papers are embedded in the text of the Chapters as published for internal use. I do not use pre-prints neither post-prints to avoid ambiguity in content. There is a public version of this manuscript, that excludes the papers, which is available online.

Chapter 2 explicitly describes the imaging pipeline for SFA technology, and all computational pipeline components are discussed. The pipeline is also discussed in its entirety, and I refer to my works on computational imaging (e.g. on demosaicing and high dynamic range) and to examples of attempts to more computer vision related processing (e.g. background subtraction from video sequences).

Chapter 3 discusses the prototyping itself and specic technical challenges, such as spectral sensitivities.

Chapter 4 proposes to discuss the illumination problem in order to take multispectral imaging outside of controlled environment.

Chapter 5 discusses selected perspectives in research, and technology transfer to industry and concludes on our research on SFA.

Chapter 6 presents an overview of my research, so that the reader can understand better the context, and how SFA is linked to the other aspects I developed. All the technical papers published are also listed there.

Chapter 7 contains a Curriculum Vitae, which states projects and funding schemes, student supervisions and so on. This also helps to understand the context of the research.

Introduction

This article was written after we gained experience on the technology, and aims at getting an overview on the imaging pipeline. It also aims at solving the problem of unbalanced spectral sensitivities identied in [Lapray 2017c] by the mean of high dynamic range imaging (HDR). The quality estimation is original and we identify the need for new research on HDR-spectral images quality.

It is published in

Discussion

Denition

We dene the imaging pipeline for SFA cameras. As shown in Figure 2.2, the pipeline is essentially similar to the CFA pipeline. Thus, it could be, in the principle, studied similarly with respect to signal and image processing theories. However, good care should be taken in several hypothesis related to the nature of the images: They are not, in general, color images, neither large band intensity images related to luminance, i.e. panchromatic.

SFA camera, similarly to CFA, samples the image domain Ω ⊂ R 2 by the use of one of the dierent sensitivities c(x) at each pixel, then M c ⊂ Ω is the subset of the image domain that is covered by the sensor mosaic. We use a similar notation as in [Thomas 2018d]. In each pixel, only one of the channels exists, so

Pipeline components

Pre-processing may typically includes a denoising process on the raw image. For instance an oset correction may be performed, where

f c M (x) = f c M (x) -f 0 M (x). f 0 M (x)
is a dark channel image. This image can be estimated by taking the median values of a number of images captured in the dark with the cap on the camera. We have not investigated further solutions to reduce the noise after the acquisition ourselves.

Several state-of-the-art techniques from greylevel or colour imaging can be adapted to this problem, and some of the denoising strategies could be deported, or performed jointly to the next blocks.

Channel balance or channel gain adjustment is developed specically in Chapter 4

where we discuss the role of illumination in SFA imaging. This is similar to white balance in colour cameras. For now, let us mention that each of the f c M (x) is typically weighted by a value that takes into account sensor sensitivities and illumination properties. By this mean, we can provide a multispectral representation that is independent from illumination. This was shown to be of major importance in colour imaging for computational colour constancy, but we demonstrate that in SFA, it is critical for acquisition itself [Lapray 2017b, Lapray 2017c], demosaicing [Mihoubi 2017b] and stability of data representation [Khan 2018c].

The task of demosaicing is to solve the problem of having a spatial shift between the spectral sampling. In other words, we want to nd the values for all the pixels in the image,

f c (x) for x ∈ Ω, such that f c (x) = f c M (x) for x ∈ M c .
And so, recover f (x) on the image domain. Although demosaicing is very important for colour imaging because of the visualisation on displays, and some standard storage formats, it is not obvious that it is always needed for multispectral data. Indeed, it seems that, for instance, texture parameters may be well retrieved from raw data rather than demosaiced data [Losson 2015]. Nevertheless, if we consider that demosaicing is a registration correction in SFA, then it is important, and if we consider measurement of a spectrum at position x, then we need to agree on the best estimate of the set of multispectral values f (x) for this specic location.

We addressed demosaicing in the PhD of Xingbo Wang [Wang 2016], in simulation, and in follow-up works on real data acquired with our prototype.

Median ltering for demosaicing raw SFA images [Wang 2013b]. The median ltering provides a safe demosaicing in the sense that it could be designed to not introduce new values in the image. However, it is quite long to compute and the visual or accuracy resulting image were not very convincing in our simulations.

Discrete wavelet decomposition demosaicing was extended to SFA [Wang 2013a]. The underlying concept of demosaicing based on wavelet decomposition is that high frequency components are similar for all the bands at the native scale of decomposition. So, the low frequency could be estimated from a higher sampled band (or a panchromatic image version), and the high frequency component estimated at a downsampled scale level can be simply reported from one band to another. The results in our paper were not very good because we used a band that was sampled as sparsely as the other ones for lowfrequencies. At the contrary, if used in a moxel arrangement such as the one from Monno et al. or on the panchromatic image, such as Mihoubi et al. did [Mihoubi 2017a], then the hypothesis may be as reasonable as for the Bayer instance. One issue that would remain is how good a SFA is to capture high frequencies when a band is occurring several pixels away from the previous occurrence. This should be related to the image content and perhaps to natural image statistics.

Xingbo Wang also developed a linear minimum mean square error (LMMSE) formulation to demosaic SFA images in [Wang 2014a]. His PhD thesis contains extended results [Wang 2016]. We improved LMMSE to N-LMMSE with Prakhar Amba and David Alleysson in [Amba 2017a]. The recent PhD of Prakhar Amba considers also more learning methods to demosaic SFA [REF PRAKHAR PHD]. Those methods performs very well and have the advantage to generalise to any arrangement of moxels. They are however constrained by the learning database, and it is not easy to predict their eciency on images that do not exhibit the same statistics.

Several other teams has also addressed this problem. Monno et al. [Monno 2015] developed several demosaicing optimal for their specic sensor. We did present a more recent state-of-the-art in [Amba 2017a] that contains references to the most recent works.

That being said, the best attempt of generalisation of demosaicing for SFA, in my opinion, is developed by Soane Mihoubi et al. in their article [Mihoubi 2017a]. Indeed the use of a panchromatic image permits the development, extension and unication of most frameworks and algorithms proposed in the literature for CFA to SFA. This work contains the methods that should be used as benchmarks when possible.

Optical setup and spectral sensitivities also impacts demosaicing. We investigated the role of chromatic aberrations in SFA camera [Wang 2014b].

Optimal pipeline should consider the joint optimisation of all the elements, such as stated by Li et al. [Li 2008] in their conclusion. This is probably the research papers that we will observe in the next years, especially with the combined use of hyperspectral database image acquisition and the use of deep learning methods. I develop this aspect in Chapter 5.

Pipeline outputs

The use of the images towards visualization must be developed. Indeed it has been common in the spectral community to either project spectral data into a colour space for colorimetric visualization or use false colours generated from three informative bands selected amongst the acquired bands or after a dimension reduction process. The colorimetric rendering has the advantage to provide a natural vision on the data, however it will hide implicitly the additional information contains in the spectral bands. The false colour rendering may contain specic informative data, but the visualisation is so unnatural that in many cases, it is hard to nd out the information visually. I would suggest that augmented rendering would provide a better compromise. This statement opens a new research paradigm, which is developed in Chapter 5.

The storage of multispectral images is a problem due to the large size of those data. It usually calls for compression. However, in the case of SFA, the raw image is not larger than a greylevel version, so storage of raw, augmented with information on the camera, e.g. moxel and spectral sensitivities, may provide an ecient way to do that. This implies that the decoder would be a little complex: Further research and standardisation should discuss those aspects.

Computer applications may take advantage of multispectral images. We could consider two ways of doing this. One way is to use the native spectral resolution of the sensor and extract information from there. This is what we have done successfully for background subtraction [START_REF] Benezeth | [END_REF]]. Another way is to reconstruct spectral radiance or reectance of the scene and use this as scene information (for material identication for instance). The last possibility permits, to some extent, a standard representation of data.

It is to be noted that, at least in the visible range, spectral images were hardly competitive versus colour or greylevel images in computer vision. This may be explained by the fact that spatial resolution was not as good in spectral images as in greylevel images, while algorithms were mostly based on gradient computation. This created an implicit dominant role of spatial resolution to the advantage of greylevel images. We can also add that dimension reduction was mostly applied before processing, then some of the advantages of spectral images were cancelled out versus a good resolution image in greylevel or in colour. This will probably change with the development of adequate processing, in particular learning protocols. However, we observe already that the use of an additional NIR channel has been accepted as valuable by the computer vision community, e.g. RGB-N or VNIR imaging.

Discussion

Historical background

This Chapter considers the realization of our prototypes. We must state that we were pushed by the community to get real data to work on SFA demosaicing when Xingbo Wang started his PhD in 2011. Back then, there were no available commercial solutions beside the rst prototypes of 4 bands, e.g. RGB-NIR realized by Ocean Optics. Most of the works were performed in simulations based on hyperspectral reectance images. This was a very reduced set of available data for us because we needed images in both visible and NIR. A time constraint was also coming from the project Open Food System in which a prototype sensor was a deliverable (See 7.8.3).

We considered several partners to help us with lters manufacture, only SILIOS Technologies could really master lter realisation at the level of a few pixels. However, We had to run for a hybrid process: It was important for us that this prototype sensed the NIR part of the spectrum for both literature consideration and project application.

Due to diculty of realisation and uncertainty of results, we decided to use directly the state-of-the-art binary-tree method to design the arrangement of moxels as a general purpose instance. In fact we realized three dierent sensor layouts, but only characterise deeply and published about one of these instances. One of the three layout is under a patent process.

One critic that we can rise here is that we should have spent more time in simulation to design an optimal design before realisation, we go back on that in Chapter 5.

Nevertheless, due to this urge, we could publish results quite rapidly, at the same time as competitive research teams. Thanks to that we had been invited to join the EU projects CISTERN and EXIST (see 7.8.2). We could also demonstrate very early image acquisitions and show the proof of concept.

Analysis on sensitivities

The core of multispectral imaging system is the set of spectral sensitivities, c(λ), which characterises the system. In many simulated works, the c(λ) are band-pass normalized Gaussian functions with means centred on specic wavelengths. In the case of SFA, this is, in general, accepted for simplicity of simulation. In practice, this is done typically to approximate Fabry-Pérot interferometers that can be realized by nano-etching of a glass substrate for instance. We investigated on the dierence between the two models in [Lapray 2017c]. The ecient part of the sensitivity may be equivalent, and most of the dierence reside in spectral area far away from the peak of sensitivity. Whether this dierence is critical for simulation is yet to be investigated.

The number and shapes of the lters is providing very dierent sensor features and performance, often depending on applications. In general, we can say that very narrow bands do not exhibit the best general performance, and an amount of correlation or overlapping between the bands benets most later computations. This statement may be wrong in the case of very specic application for which sensing a set of narrow parts of the spectrum is useful, e.g. some biological measurements. However, notch lters are not better either, especially because they would increase the contribution of noise to the signal.

We discussed the diculty on the choice of lters, layout and number for the very general cases in the following perspectives:

• Filters shape and number for spectral reconstruction [Ansari 2017, Wang 2014c, Wang 2013c].

• Filters shape and number for colorimetric estimation [Wang 2015].

• Filters shape and number for illuminant estimation [Khan 2017c].

In fact, the diculty of lter realisation makes the practical instances quite far from the ideal case or from the simulations. As an example, measurement of the sensitivities of our prototype sensor are shown on Figure 3.2, while simulations and lter measurements are shown in Figure 10 of the following article.

We can observe that the lters are not of any ideal shape. In addition, we can observe that there might be curves that are not uni-modal. This is true for most RGB-NIR instances, for most commercial instances such as IMEC sensors, and so on. Although the general shape is not critical for later computation, the mix of very dierent spectral components into a single channel may be a problem for the applications, and specic solutions have been considered. We considered the special case of our prototype and proposed to demultiplex visible and NIR components [Sadeghipoor 2016]. This problem is generally addressed in the literature as visible-NIR separation and applied to RGB-NIR sensors.

How to design a sensor?

I would not discuss the spatial distribution in this manuscript, many ad hoc solutions have been proposed. It is most likely that similarly than for CFA, the best instance would be a pseudo-random arrangement. But this would depend on the sensitivities.

Due to the many dierent possibilities, an exhaustive search would be very dicult and it is dicult to nd a model that gives all freedom in all the pipeline elements.

However, we recall with Li et al. [Li 2008] that the pipeline must be optimised in its entirety despite of the diculty, if not, the local optimisations would only relate to academic niches. A recent article in this direction proposed a solution for natural scenes [START_REF] Li | [END_REF]]. They observed an optimal spatio-spectral behaviour with 5-6 bands to describe the visible part of the spectrum. I will go back to this point in Chapter 5.

One question remains: Should we try to achieve a universal design that respect general natural image statistics or to dene specic sensors for particular uses? In fact, it would be easier to dene an optimal SFA camera for a particular application where the cost function is very well understood, or at least computed. However, the market mass of each of those instances would probably not be sucient to see many instances to appear in the market. An ubiquitous instance that permits several type of applications would lead to cheap and robust options. It however must outperforms other existing solutions.

And it would require the tuning of the pipeline for each specic uses. This is also further addressed in Chapter 5.

Article

The paper is here -as published.

Pierre-Jean Lapray, Xingbo Wang, Jean-Baptiste Thomas and Pierre Gouton.

Multispectral Filter Arrays: Recent Advances and Practical Implementation. This article is very recent, so any analysis on how it has been read is irrelevant.

Discussion

According to Eq. 1.2, for each specic e(λ), f (x) have good chances to be very dierent. It is intuitive that for most tasks, it is useful to get f (x) related to r(x, λ), with no inuence of e(λ). We propose that for any e i (λ), there would be a transform G s i that computes the f i (x) into a stable representation f s (x), which can then be related to r(x, λ) thanks to a calibration process. We write this in the very general case in Equations 4.1 and 4.2. In our research, the transform is a linear transform to investigate the concept, but any formulation may be investigated in the future.

f s (x) = G s i (f i (x)) (4.1) f s (x) = G s i r(x, λ)e i (λ)c(λ)dλ (4.2)
We dene that as spectral constancy [Khan 2018c]. This is very similar to computational colour constancy, and similarly, a class of solutions assumes that the illumination e i (λ) is known. In practice, the illumination estimate in the sensor domain is enough [Khan 2017c], and we show that a linear transform is accurate enough to improve greatly the representation of spectral data [Khan 2018c]. We also show that a linear diagonal Chapter 4. Illumination transform, equivalent to a Von Kries transform, is not enough for most practical choices of c(λ), but would benet from a spectral adaptation transform. This later concept is similar to a chromatic adaptation transform [G. 2004], or to spectral sharpening [START_REF] Finlayson | [END_REF]].

The article in next section addresses this problem by providing experimental results in simulation. We used an evaluation based on spectral reconstruction, which permits to compare between dierent set of c(λ). We show also that the results depend strongly on a good illuminant estimation, but we have not been able to quantify yet how well it should be estimated to become useful. Validation of the simulation results in practice is an ongoing work. Upcoming communications will demonstrate how robust the proposal is, and how good must be the illumination estimated for material identication based on spectral reectance.

Development of this concept by the use of dierent technologies, from the investigation based on the dichromatic reectance model to deep learning optimisation of the imaging pipeline will be addressed in the future.

Article

The paper is reproduced here -as published.

Research

This Section provides analysis and guidelines for further research in the academics.

Use of improved imaging model and diverse modalities

The use of the dichromatic model [START_REF] Tominaga | [END_REF]] for computational spectral imaging is one serious research direction. Traditionally, in our scientic community, the diuse conditions were enough when we had no high dynamic range capture and rendering devices. At the instar of the computer vision community that has used the dichromatic reectance model historically, in the computational colour or spectral imaging, this model have been seldom used beside of computational colour constancy. The reason is that we intended to capture radiance for photography. If we shall take into account more and more the separation between illumination and object properties, this model could be considered in most parts of the imaging pipeline. Of particular interest, it could create a continuous link to the eld of material appearance and computer graphics and relate imaging technology to the Bidirectional Reectance Distribution Function (BRDF). And by extension the imaging model used must relate to the general case of the radiative transfer and advanced material descriptions.

On another direction, similarity and dierences with approaches used in remote sensing would be very interesting. There are many possible extensions by revisiting spectral unmixing under dierent priors for close-range multispectral imaging, e.g. separation between visible and NIR components.

Other additional modalities, such as polarisation, may be considered into the SFA paradigm, which may then become lter arrays imaging. Database of joint spectral and polarimetric images started to be developed [Lapray 2018]. 3D, or at least depth, could also be considered.

Unication of pipeline

As mentioned in Li et al. [Li 2008], for colour imaging, demosaicing is now only an academic niche if taken outside of the full pipeline. This is not yet the case for SFA, but soon will be when we will reach a global understanding of the technology. This statement is also general for all components of the SFA imaging pipeline, including superresolution and denoising that are not very well considered in this manuscript. We observed in several cases that the spectral sensitivities of the camera impacts and conditions the choice and performance of computational imaging algorithms. It is therefore very important to make choices that are compatible and optimal. One very important task for that is to dene what is a multispectral image. Indeed, the colour imaging case is very well dened as the capture of an accurate or pleasant colour image. The hyperspectral case is also fairly well dened as a spatial, spectral measurement and is treated or considered like this in the literature. The case of multispectral is however ambivalent due to the multiplicity of possibilities and abstract meaning of the values for each band. Some authors consider it from an accurate colour imaging instance and use a colour transform, some consider it as a mean to estimate a spectral measurement image and use spectral reconstruction in order to transform the data into the well understood context of spectral measurement. Most people are working on those two grounds. In my opinion, the case of multispectral could, of course, be considered as mentioned. But it is not obviously necessary. It is possible to only consider the data from the sensor and to use it as abstract information for any task. Within this general context, the two former proposals are only dierent and specic applications that use those raw SFA data. This has not been done much yet because the quantity measured are abstractions. Thus it has been more convenient to attach the SFA concept to color or spectral measures. I believe this is changing now that there are commercial instances which are investigated by the computer vision community rather than the imaging community. We can note for example: [START_REF] Benezeth | [END_REF], Neuhaus 2018, Winkens 2017].

Computational models and methods

Nevertheless, the optimisation of the SFA imaging pipeline is required. So far, only general signal processing cost functions, such as PSNR, have been proven ecient for optimisation due to versatility, diversity and abstraction of values. In general, natural or laboratory hyperspectral images have been used to simulate and evaluate the pipeline, coupled with the imaging model of Equation 1.2.

Models to optimise are often based on the simulation of part of the pipeline or pipeline item . We observe several dominant approaches in our community. A good part of the literature considers linear formulations to optimise the resulting image based on simulations, in particular, denoising, demosaicing and spectral sensitivities (e.g.

[Zahra [START_REF] Sstrunk Zahra | Optimum spectral sensitivity functions for single sensor color imaging[END_REF]). Some authors propose to use variational models to process images between raw data and the nal colour images, in particular image enhancement, tone-mapping, white balance, etc. (e.g. [START_REF] Provenzi | [END_REF]). Recent literature using deep learning to optimise the color imaging pipeline [Zhou 2018[START_REF] Henz | [END_REF]] opens the door to further studies and new optimisation frameworks.

It is hard to recommend the use of one model compared to another for further studies.

To me, the linear formulations permit a good tracking of errors and permit to decompose each steps easily, however they are not providing the best possible peak accurate results in most cases though they are robust and relatively fast. The variational methods are very interesting in the sense that they provide a good model for images that takes into account local edges, and they are embedded into a robust mathematical formulation [START_REF] Provenzi | Computational color science[END_REF]]. Problem is that the inversion of the model is usually iterative and thus it may not be feasible in real-time though it generally converges fast, which may or may not be a very important feature for pipeline optimisation. Deep learning formulations may nd out unexpected optimal solutions or validate empirically traditional hypothesis. It is promising in a sense that we may nd interesting hypothetical explanations by understanding how the solutions were identied. It is probable that all those approaches should be used and compared.

It is yet not sure how the need of data will be handled. The deep learning techniques would require many data, and the community has been looking for such data for decades.

On the other hand, there are several databases that are published and made available in the recent years thanks to progress in optics and the commercialisation of hyperspectral cameras, such as [START_REF] Hsi | HySpex VNIR-1800[END_REF]].

Any optimal proposal will be limited to the choice of a cost function, and there is a very important and dicult scientic challenge in this direction.

Visualisation and image quality

Quality index

One necessary condition for pipeline optimisation is an objective measure for quality. In the case of SFA in the visible and potentially NIR, we do not have any very good measure, and the PSNR has been the most robust indicator so far. We have recently started to investigate on that aspect through the Master thesis of Nathan Miot-Battu 7.7.3, and started to use no-reference image quality metrics by image band. We did not push the investigation far enough, for instance we have not re-trained the metrics from Bovik and his team, neither tried to correlate the visual sensation generated by one arbitrary spectral band to the metrics. Nevertheless, the result of our reection is:

• If the images are converted to colour images, then we can use colour image quality indexes.

• If the SFA individual bands are considered as greylevel images, there is no evidence that those images follow the same assumptions or statistics than the large band or photometric greylevel images. This needs further investigations.

• If we should reconstruct spectra, then there are also problems in the evaluation of the quality of the measure. Today's measures from PSNR, GoFC or Angular error are highly correlated and usually only provide relative indicators and no spatial information. One very interesting direction for this problem is the recent works of Noel Richard and his team to provide the mathematical formulations that can be used to describe spectral images. However, those tools are not yet adapted to multispectral images.

• Another indicator of quality for a pipeline is the results for a specic application.

However, good care have to be taken in this direction: Final application evaluation may also introduce bias. Classical way to annotate databases may not be the most adequate: We annotated manually videos [START_REF] Benezeth | [END_REF]], but could only label what was visible: it is possible that we missed some information that actually the machine may see.

Data visualisation

This leads us to the visualisation of the data. I will develop now the concept of visualisation of multispectral images, and this may be related to quality, and later to video.

The most natural way to look at a multispectral or hyperspectral image is to have it to look as a colour image. This is usually performed by the application of a colour transform, which could be respecting colorimetric principles or not. Typical colorimetric transforms use integration on CIE colour matching functions or equivalent formulations.

The less advanced methods would simply choose three bands corresponding to red, green and blue. One widely used visualisation of spectral images is the visualisation in false colours, which in many cases appears very limited to me. Although in terms of computer vision, an abstraction can be represented as a magnitude value on one channel, but the combination of these channels into a colour code seems not natural to me. I may have missed something in the literature because I did not really identify the advantage of this proposal. In addition, the projection on 3 bands, colorimetric or not, necessarily reduces the amount of information, whatever the technique used. It seems not optimal to use only a visible 3D (or 5D if we consider the spatial dimensions) projection of this huge amount of data, however, it could provide a natural feeling to the observer. In a pragmatic set-up, the data is most likely used to extract specic information, then the natural image may serve as a basis, reference or support, but the information extracted may be semantic or invisible directly (e.g. dyes quantity on a paint or ink properties extracted by unmixing). This information should be visualised in some way. Instead of looking for a better projection onto three components, I suggest that we look rst for the most suitable colour image, which could be either colorimetric, or the most natural, or the best look. Then, we can augment this image with pertinent information extracted from the multispectral data. The fusion of those two sets of information is anticipated to be dicult, and can use research on transparency, but also ergonomics and so on.

We must search for ways and models that permits to fusion those data. One natural direction may be to use semi-transparent object or information that overlaps with the natural scene. In order to be natural, not disturbing but still visible, the colormap may depends strongly on the background and situation, which creates very strong constraints.

There is an open space for research toward this direction, and a nice link can be made with my new research on transparent material perception.

Spectral video

In this case, Equation 1.1, becomes Equation 5.1, where t stands for time.

f (x, t) = s(x, t, λ)c(λ)dλ (5.1) This equation can be considered from dierent perspectives. One way to understand it is to consider spectral video, where the time is another dimension that opens to realtime applications and new classes of processing and algorithms. Another way to consider this equation is to consider that t -1 and t + 1 frames can be used to better process the t frame and produce a better image. This is what we have done for HDR spectral imaging.

This is what is done in recent smartphones and some cameras. There is however only little academic literature that considers explicitly this aspect for SFA, so there is an open space for research there.

The former proposal opens the problem of handling of spectral video and also could be developed in two directions: Either a human is the nal user or the machine is. In the last case, the machine will need to extract features that helps it to interact with its environment and realize the task it has been made for. This would surely call for feature extraction and classication and should not surprisingly be treated from a machine learning perspective in the next years.

If a human observer is the nal destination, then the rendering of the scene should be meaningful according to the human cognitive system. Then I would suggest to consider it as an augmented reality problem, where the scene is rendered in a natural pleasant way, so that the person is not disturbed by the content, but in the same time good use should be made of the extra, pertinent information. This extra-information should be added to this natural content in a way that it is not disturbing, but full of sense. It is a similar case as above, but more complex due to the change of scene over time.

Standardisation

Standardisation of sensors is "to dene a unied method to measure, compute and present specication parameters for cameras and image sensors used for machine vision applications". This is also a quality assessment. There is an attempt from the European Machine Vision Association [EMV 2018a] toward sensors standardisation. EMVA 1288 in particular, [EMV 2018b] is meant to be extended to SFA cameras and sensors. Reaching an industrial quality and communicating about it is indeed very important for the large spread of those type of sensors.

An other standardisation attempt is to dene multispectral image formats, which is dicult due to the diversity of bands, number of bands and various acquisition techniques, it is very important that the le formats also contain addi- These initiatives are very important, although in general those collective works are very time-consuming and sometimes not very well acknowledged in academic researcher evaluations. They are of specic importance in this case because industry and users need those guarantees and tools to accept further technology transfer.

Technology transfer

This Section addresses the part of the academic research that could be converted into innovation. The OECD denes innovation as "An innovation is the implementation of a new or signicantly improved product (good or service), or process, a new marketing method, or a new organisational method in business practices, workplace organisation or external relations." [START_REF] Oec | OECD-innovation[END_REF]]. The business dictionary denes it as "The process of translating an idea or invention into a good or service that creates value or for which customers will pay." [BD-2018].

In the context of SFA, I would in general qualify the innovation of evolutionary, because the technology permits a continuous evolution of machine vision. However, in some cases, the quality of the information extracted (i.e. beyond vision) may permit revolutionary innovation in some applications (e.g. as it is the case for crop monitoring with the joint use of visible and NIR information).

In terms of timing, innovation on, or based on, SFA technology is really timely due to the general need of methods to improve automation and the recent data science development. In Europe, there are several facilities and funding that may encourage innovation in this direction, which is a very good sign for our research domain. We could see for our concern the projects EXIST and CISTERN (Chapter refchap:cv).

General applications

Applications in computer vision (applications in medical, security, robotics, autonomous driving, etc.) is one of the more prominent opportunity for SFA. Indeed, it is one of the technology that permits spectral imaging to get out of the laboratories. It permits to capture additional data while still using a very similar pipeline than the usual imaging techniques. However, there is the need to demonstrate breakthrough in applications.

Adequate algorithms that compete or overcome the state of the art are needed, but they must be embedded into demonstrators and not limited to simulation or laboratory prototypes.

In addition, available commercial solutions are not yet excellent and users need expert to help them to develop solution. So there is a room for expertise transfer and consultancy. This shall improve in the next years.

Other applications to material appearance measurement are also very appealing.

Technical commercial products

Typical products that may develop around the SFA technology may be the optimisation and ne tuning of the imaging pipeline for a specic application or costumer, and based on a commercial solution that may be more or less stable. The products that could be developed may be implemented into a software (API or SDK) that could answer the need of a consumer. At another level, the embedding of those solution into hardware and real-time solution may be considered. Video codecs and visualisation modules could also be a part of these solutions.

There is room for the creation of spin-o companies into these directions now that sensors are available. It is to be noted also that several new users of SFA have a problem of expertise to use the captured raw data for their application. We observed that recently in the medical imaging community when a meeting of SFA users was organised in Cambridge to discuss the dierent understanding and results within a group of similar medical expertise.

Conclusion

This research permits to demonstrate in practice a technology that only existed in simulation. It contains a set of prototypes of camera, experimental data, algorithms and methods that permit to shape or use the data captured for machine vision. Based on this proof of concept, here is room to develop optimal or more adequate algorithms and sensors. There are also interesting research directions and perspectives that could generate activities for the next years. Interestingly, we note that several aspects could be shaped as a commercial product if the sensor industry manages to develop standards and commercial oers, which they seem to do. • We proposed to use image contrast measure to correlate with gloss perception [Thomas 2017a].

• We initiated also more qualitative investigations by creating a collection of real objects that permits to investigate in practice the concept of material appearance [Thomas 2018c, Thomas 2018a]. A preliminary analysis has been submitted to the Colour and Imaging Conference [Gigilashvili 2018b]. Dierent actions are taken at this moment to use those objects.

Image acquisition -Camera

• Since 2010, I investigated on color and multispectral acquisition and associated signal processing. On color acquisition, I did transfer my expertise from displays to scanners [Thomas 2011] and I co-wrote a book chapter on color cameras [Nozick 2013a, Nozick 2013b].

• On multi-spectral acquisition, I develop the SFA (Spectral Filter Arrays) technology [Thomas 2014c, Thomas 2014a, Thomas 2014b, Thomas 2016a]. This was concretised by the short course we gave at the Colour and Imaging Conference [Thomas 2017c] and by the invited talk at a Dargstuhl Seminar [Thomas 2018b].

• We realized a prototype of a joint visible and NIR camera [Lapray 2017a, Thomas 2016b, Lapray 2014b, Lapray 2014a] and redene the imaging pipeline [Lapray 2017b, Thomas 2017b]. This was possible via the funding of OFS and a BQR ?? and with the help of a Post Doc Pierre-Jean Lapray.

Data are very important and we generated several databases for multispectral and hyperspectral imaging benchmark or simulations [Lapray 2017a, Lapray 2017b, Khan 2018a].

• We discussed what may be optimal sensor spectral sensitivities [Wang 2014c, Peguillet 2013, Wang 2013c, Lapray 2017c, Ansari 2017]. We investigated demosaicing and related aspects, in particular through the PhD of Xingbo

Wang [Gigilashvili 2018a, Mihoubi 2017b • A demonstration of multi-spectral video applied to background subtraction has been shown [START_REF] Benezeth | [END_REF]].

I do believe that the simplicity of the SFA concept coupled with illuminant understanding is the key for getting multispectral cameras out of the labs.

Visual aspects and quality

In parallel, I am developing a research on vision and on quality linked with the above two aspects. I think these are the most fundamental points of my research and it is a long shot strategy.

• I considered the gamut of an image and the sampling of color spaces [Colantoni 2016, Thomas 2013a, Thomas 2007c, Thomas 2007a]. Through the notion of structures in an image and the notion of graphs, we proposed the visualization of image data in a new way [START_REF] Colantoni | [END_REF]].

• More recently, I contributed to evaluate the perceived quality of a displayed image through the PhD of Ping Zhao [Zhao 2015a, Zhao 2015b, Zhao 2014b, Zhao 2014a[START_REF] Zhao | [END_REF]]. We considered the use of a camera to replace the observers in the conduction of quality evaluation experiment.

Publications

Publications are listed below. I refer to my Google Scholar prole for indexes and citations 1 . There are several submitted works that do not appear at those links. You may refer to my personal webpage for accessing my publications 2 .

On Figure 6.2, we can observe the number of articles I published by year between 2007 and 2017. We can observe the dierent consequences of my actions: less publications after I took my position at UB, increasing number after I obtained the funding for the OFS project, and increasing numbers after I was out of teaching duties. On the numbers provided and due to the revision time, this graph must be smoothed, and the most interesting information is that I demonstrate a scientic production that is increasing thanks to experience and funding, coupled with my délégation CNRS and my détachement.

On Figure 6.3, I am showing the number of citations of my publications according to google scholar. We can observed that after our paper presented in Chapter 3, there was an exponential increase. This is the consequence of the research I performed on SFA, as exemplied by the invited talks.

1 https://scholar.google.fr/citations?user=MkzII3cAAAAJ&hl=fr 2 http://jbthomas.org/publications-2.html • In this context, my teaching is associated to the department IEM (computer science, electronics and mechanics) and my research is associated to the Le2i (laboratory of electronics, computer science and image).

• I was a 50% invited researcher at the IVRL, EPFL (Lausanne, Switzerland) on year 2015-16 in the context of a délégation CNRS. We developed a method to separate visible and NIR contributions to the image captured by our sensors [Sadeghipoor 2016]. I also enjoyed this year to visit and initiate collaborations with my colleagues at LISTC and LNPC.

• My expertise lies in color imaging from acquisition to reproduction, through technological aspects, physical measurements and human visual system understanding. My teachings include but are not limited to signal and image processing, sensors, color science, colorimetry, color imaging and multispectral imaging. • I was principal researcher and coordinator for the project PSPC Open Food System 2 for my lab.

• I was technical coordinator for the EU projects H2020-EXIST3 and CATRENE-CISTERN 4 .

• I was co-head of the team MOTI (Methods and tools for image processing) of my lab in 2015 and I was elected to seat at the Lab council between 2012 and 2016.

• I supervised and am supervising several PhD and Master students.

• Industrial references may be available under conditions.

7.6 Teachings • I also gave an image processing course in English language within an international Master, MaTEA, at Agrosup Dijon. At Fall 2016, the Master opened with 14 students. Along with this Master, we signed an MoU with HAINAN UNIVERSITY in China, in which we agree on exchange of Master students. We also did collaborate with the French Ambassy in Nigeria and our students from there were attributed 3 grants from oil industry to come to study with us. I had to leave for my sabbatical, so I handed over the Master program to Professor Jean-Marie Bilbault in September 2016. Together with Marius Pedersen (NTNU-Gjøvik), we obtained a 1 year traveling grant within the AURORA, PHC call. We worked on the orientation selectivity in chromatic contrast sensitivity of the human visual system and its consequences on display quality.

We wrote and managed the project together for our respective Universities. Within the PARI there were several call for projects. The project that contains the thesis of Xingbo Wang was co-written by Pierre Gouton and me. The project that contains the thesis of Haris Ahmad was co-written by Olivier Laligant and me.

BQR PRES 2014

We obtained a local funding for investigating and developing the use of multi-spectral cameras in automotive applications. This funding permitted to duplicate prototypes of Spectral Filter Arrays cameras.

Pierre Gouton managed this project that we wrote together. I only participated lightly to this project as external member. Pr Alamin Mansouri was in charge for this thematic at the Le2i.

Summary

A timeline summary is shown in Table 7.5, where the reader will observe the link between the projects and research fellows. Jessica El Khoury was funded by the OFS project, as well as Pierre-Jean Lapray. Xingbo Wang and Haris Ahmad were funded by the regional council through the PARI, the co-funding came from NTNU for both of them. Ping Zhao was funded on the Hypercept project at NTNU, and we could develop his mobility via

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Multispectral imaging . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Scientic publications on multispectral imaging . . . . . . . 5 1.4 Spectral lter arrays . . . . . . . . . . . . . . . . . . . . . . . 7 1.5 Overview on related research on this technology . . . . . . 8 1.6 Overview on scientic contributions related to this manuscript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.7 Structure of this manuscript . . . . . . . . . . . . . . . . . . . 10 1.1 Context The core of the research presented in this manuscript has been developed at the Le2i, Laboratoire d'Electronique, Informatique et Image (UMR CNRS 6306, then FRE 2005), at UFR Sciences et Techniques of Université de Bourgogne, then Université de Bourgogne, Franche-Comté. The time lapse considered is seven years between 2010 and 2017. What is presented is a reduced set amongst several research activities, see Chapter 6 for a more comprehensive overview.

(

  IVRL) at the EPFL in Switzerland. The latter collaboration was permitted by a Délégation CNRS in 2015-16. The former is a historical collaboration developed by several research stays, of which a long stay of 4 months in fall 2012 (permitted by capitalization of teaching hours) and a Mise en détachement since Fall 2016, which continues to Fall 2019.

  Due to the size of the community concerned by this eld, the impact factors for the relevant specialised journals is not super high compared to other elds. Traditionally, the community of color or multispectral imaging have published in journals such as Wiley's Color Research & Applications [IF=0.798], Journal of the Optical Society of America A [IF=1.621], IS&T Journal of imaging Science and Technology [IF=0.35] and IS&T -SPIE Journal of Electronic Imaging [IF=0.754]. Works related to image processing have sometimes been pulished in IEEE Transactions on Image Processing [IF=4.828], Elsevier Signal Processing: Image communications [IF=2.244] and Image and Vision computing [IF=2.671]. Works related to Vision have been sometimes published in ARVO Journal of Vision [IF=2.671]. Works related to multispectral imaging have sometimes been published into other neighboring communities, such as in MDPI Sensors [IF=2.677], SPIE Optical Engineering [IF=1.082], IEEE Transactions on Geoscience and Remote Sensing [IF=4.942], ACM Transactions on Graphics [IF=4.096]. It is noteworthy that with the acceptance of the colour images for computer vision applications, many experts in colour imaging turned to image processing or computer vision journals. Computational imaging has been a growing eld, and we have seen the creation of new journals in the recent years, such as IEEE Journal of computational imaging and MDPI Journal of Imaging created both in 2015, IS&T Journal of perceptual imaging created in 2017.

Figure 1 . 1 :

 11 Figure 1.1: Graph representing publications by eld that are concerned by the keyword multispectral imaging from web of science, accessed on May 30, 2018.

Figure 1 . 2 :

 12 Figure 1.2: Graph representing the number of publications with the keyword multispectral imaging from web of science, accessed on May 30, 2018.

Figure 1 . 3 :

 13 Figure 1.3: Graph representing publications by elds that are concerned by the keyword colour imaging from web of science, accessed on May 30, 2018.

Figure 1 . 4 :

 14 Figure 1.4: Graph representing publications by elds that are concerned by the keyword hyperspectral imaging from web of science, accessed on May 30, 2018.

Figure 1 . 5 :

 15 Figure 1.5: Graph representing a typical instance of the spectral lter arrays technology.

Figure 2 . 1 .

 21 Figure 2.1. It has been self-cited twice but is very recent.

c

  M c = ∅. In general, all pixels are represented in one channel such that c M c = Ω. The pixel values of the mosaiced image are denoted f c M (x) for x ∈ M c .

Figure 2 . 1 :

 21 Figure 2.1: Graph representing access to this article on the publisher website on May 30, 2018.

Figure 2 . 2 :

 22 Figure 2.2: Graph representing a typical instance of the spectral lter arrays imaging pipeline.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

 17

3. 2 . 1

 21 Historical background . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.2 Analysis on sensitivities . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2.3 How to design a sensor? . . . . . . . . . . . . . . . . . . . . . . . . 19 3.3 Article . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.1 Introduction This article was written after we had realized the lters of our SFA prototypes, but before full integration into a camera that can capture images. It contains the state of the art for snapshot spectral imaging and denes practical instances of SFA. It is referred to in the literature for the general denition of this technology and for showing one of the rst published implementations, but also for the state of the art. It is published in Sensors, MDPI, which is a fast track, online open access publication Journal. The journal has an Impact Factor of 2.677 (2016) and a 5-Year Impact Factor: 2.964 (2016) The H-index of this journal by Scimago is 104 and is in the Q2 quartile for electrical and electronic engineering. The JCR category by web of science is Q1 for instrumentation.This article has been very well read according to the statistics shown by the Journal in Figure3.1. According to google scholar, it has been cited 65 times onMay 30, 2018, which is rather good for our eld. Amongst the references, I cited it 11 times. It has also been cited by the competitive teams introduced in Chapter 1. It is to be read together with a subsequent article published in the same journal[Thomas 2016b], which presents the spectral characterisation of the nal camera.

Figure 3 . 1 :

 31 Figure 3.1: Graph representing access to this article on the publisher website on May 30, 2018.

Figure 3 . 2 :

 32 Figure 3.2: Graph representing the spectral sensitivities of the prototype sensor.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.3 Article . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.1 Introduction This article is the theoretical core of the PhD of Haris Ahmad, and the results are based on simulation. Although it addresses general spectral imaging, the concept is very useful in the case of SFA, where the camera maybe used in uncontrolled illumination environment. The thesis is still ongoing, so this is a work in progress. Maturity is still required on this topic. In particular, we are currently investigating the concept in practice for material identication. Further research must address spatial variation of illuminations and shadows or object interaction. It is published in Journal of Imaging Science and Technology from IS&T, which is a focused journal for our community. The journal has an Impact Factor of 0.35 (2016) and a 5-Year Impact Factor: 0.46 (2016) The H-index of this journal by Scimago is 37 and is in the Q4 quartile for the topic of the article. The JCR category by web of science is Q4 for Imaging Science & Photographic Technology.

  tional information and not only pixels values. The Division 8, TC 8-07, of the CIE addresses this issue in the Technical Report Multispectral Image Formats [CIE (International Commission on Illumination) 2017].

6. 2

 2 Color reproduction -DisplaysBefore 2010, I focused on colorimetric characterization of displays.Starting from physical modeling, then shifting to spatial aspects, edge-blending and 3D, I developed an expertise on physical colors and perceptual colors. I did communicate and still do on this topic[START_REF] Pedersen | [END_REF][START_REF] Colantoni | [END_REF], Thomas 2010b, Thomas 2008c, Thomas 2008a,

Figure 6 . 2 :

 62 Figure 6.2: Graph representing publications between 2007 and 2017. In blue conference proceedings, in yellow journal articles.

Figure 6 . 3 :

 63 Figure 6.3: Graph representing references to my work according to Google Scholar on May 29, 2018.

  due to the IEM department of the UFR sciences et techniques at UBFC-Dijon.• I had the responsibility of the module colorimetry in the rst year of Master EVA, in which I built a new course format. I handled lectures, TDs and TPs. I initiated a project-based process with them, in which they were responsible, with my help, to design a topic to work on and present to the rest of the class. I also invited several academics 5 , from Norway in particular through our ERASMUS agreement or other research projects; Who gave a few hours teaching or participated in the recommendations for their projects. I also received colleagues from France to participate. This specic format permitted to make the students more responsible for their global studying project, and according to their evaluation and feedback, they enjoyed this format.5 Pr. Ivar Farup, Ass. Pr. Marius Pedersen, Edoardo Provenzi, Philippe Colantoni.• Beside this responsibility, I was participating to colleague's courses with dierent implications depending on the years and service needs. Noticeably, I was involved in: Electronics TP in L1 and L2; Introduction to vision TP in L2; Professional project TP in L1; Signal processing TP in L3; Image processing in M1 (part TD, TP, CM); Spectral imaging course in M2 (part TD, TP, CM).

  the Conseil Régional de Bourgogne permitted to nance 2 PhDs, the PhD of Xingbo Wang and the PhD of Haris Ahmad. Both co-funding came from NTNU-Gjøvik in Norway.

7. 8 . 8

 88 BQR 2012 I obtained a local funding for developing the thema of Technological obsolescence in contemporary art: The case of experimental cinema of FLICKER. This was an echo to my post-doc at the C2RMF. I wrote and managed this project. 7.8.9 Hypercept I was invited to participate to the hypercept 7 project funded by the Norwegian research council. This project provided me the possibility to give continuity to my historical collaboration with HIG/NTNU-Gjøvik. I only participated in this project as external member. This helped to nance many travels and stays at NTNU. 7.8.10 COSCH I am a member of the network COST project COSCH 8 dedicated to cultural heritage.

Table 1 .

 1 1: Gantt chart putting into perspective projects, supervisions and publications directly related to SFA in a timeline. Numbers of publications refer to Chapter 6 labels. A similar table with research comprehensive content is available in Chapter 7, where the reader will nd a description of the projects and more information. This table is made in early June 2018.

	7 years on SFA

  Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2.1 Denition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2.2 Pipeline components . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2.3 Pipeline outputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

	Chapter 2 Imaging pipeline
	Contents
	2.1

2.3 Article . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

  Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5.1.1 Use of improved imaging model and diverse modalities . . . . . . . 23 5.1.2 Unication of pipeline . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.1.3 Visualisation and image quality . . . . . . . . . . . . . . . . . . . . 25 5.1.4 Spectral video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.1.5 Standardisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

	Chapter 5 Perspectives and conclusion
	Contents
	5.1

Haris Ahmad Khan, Jean-Baptiste Thomas, Jon Yngve Hardeberg and Olivier Laligant. Spectral Adaptation Transform for Multispectral Constancy. Journal of Imaging Science and Technology, vol. 62, no. 2, pages 20504-120504-12, March 2018. 5.2 Technology transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.2.1 General applications . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.2.2 Technical commercial products . . . . . . . . . . . . . . . . . . . . 28 5.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

  Material appearance . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 6.2 Color reproduction -Displays . . . . . . . . . . . . . . . . . . . . . 31 6.3 Image acquisition -Camera . . . . . . . . . . . . . . . . . . . . . . 31 6.4 Visual aspects and quality . . . . . . . . . . . . . . . . . . . . . . . 32 6.5 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 6.5.1 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 6.5.2 Conferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 6.5.3 Book chapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.5.4 Noticeable talks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 My research focuses on colour and multispectral imaging. I tend to investigate more and more the concept of material appearance in a complex scene from physical estimates or measurements of their surface properties. A graph that summarize the relationship between the research topics in link to the projects, publications, collaborations and people is shown on Figure 6.1.Figure 6.1: Graph representing the research topics in link to the projects, publications and people (PhD students and Post-Docs). The graph shows also collaborations. Physical proximity to research keywords is roughly representative to the link between the projects and the eld.6.1 Material appearanceVisual appearance is a very broad concept. Although most people can perceive and describe the appearance of objects, it is not yet understood how, and what measure we can give to this percept. This is a trans-disciplinary research eld, where I try to use imaging techniques to infer object appearance properties.

	Chapter 6 Research summary and communications
	Contents
	6.1

•

  We consider the dehazing aspect of image in particular through the[START_REF] Khoury | [END_REF][START_REF] Khoury | [END_REF][START_REF] Khoury | [END_REF], El Khoury 2014].

	PhD	of	Jessica	EL	Khoury	[Cuevas Valeriano 2018b,	El Khoury 2018a,
	Cuevas Valeriano 2018a,	El Khoury 2018b,	de Dravo 2017a,	de Dravo 2017b,
	• We investigate on illuminant estimation from uncalibrated multispectral images
	and dene a stable spectral representation based on spectral constancy through the
	PhD of Haris Ahmad Khan [Khan 2018c, Khan 2018b, Khan 2017b, Khan 2017a,
	Khan 2017c, Thomas 2015].

  I am currently Postdoctoral research fellow at NTNU-Gjøvik, working on the MUVApp project. This project is dedicated to understand and measure visual appearance of objects (See 7.8.1).

	• I am also Maître de conférence (Associate Prof.) at Université de Bourgogne,
	Franche-Comté (Burgundy, France) although I am in a sabbatic leave until 2019
	(Mise en détachement 2016-19).

6.5.1 Journals

Impact factors for 2016 are provided when applicable, and number of citations from google scholar at early June 2018.

1. Jean-Baptiste Thomas and Ivar Farup.

Demosaicing of Periodic and Random

7.2 Synopsis

•

Table 7 .

 7 7.6.2 HoursI did give teachings every year, except when I was in délégation CNRS, for which UBFC received a nancial compensation. Since I am in détachement/sabbatical now, I do not have any teaching duty at UBFC. Note that I had to refuse my PEDR prime, obtained in 2016, because of this situation. 1: Synthesis of teaching hours by year.7.6.3 Responsabilities• I took the responsibility of the Master Advanced Electronic Systems Engineering 6 , taught in English, in Spring 2015. My main action was to start this Master program.

	Academic year 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16	2016-17
	Hours eq. TD	226	231	210	231	243	CNRS	Détachement

  • I do not have teaching duty during my détachement, however I occasionally give lectures to the Master student within the ERASMUS MUNDUS COSI and international 3DMT Master programs. I also participate to the ERASMUS MUNDUS COSI program as faculty advisor or at the quality board. Within this role, I participate to the dierent meetings of the consortium, quality evaluation and to the welcome of the new students and Master thesis defence and graduation ceremonies.My expertise in educational management is thus very oriented toward International or EU joint degrees and to the management of a specic corpus of students with specic needs and in collaboration with the International Oces of the Universities. worked with 2 Post Doctoral fellows that we recruited on projects OFS and EXIST, which are summarized inTable 7.2.

	7.8.5 AURORA 2015
	7.7 Supervisions
	7.7.1 Post Docs

I

Table 7 .

 7 AURORA project. Keivan Ansar was funded by the EXIST project. My dierent research stays were funded by several projects, when related. 5: Gantt chart putting into perspective projects and supervisions in a timeline. This table is made in early June 2018.

	7 years on SFA

7

http://colourlab.no/research_and_development/research_projects/hypercept 8 http://www.cost.eu/domains_actions/mpns/Actions/TD1201 the

vol.14, no. 11, page 21626, 2014.

http://www-iem.u-bourgogne.fr/MASTER/MSCAESE/homepage_128.htm

http://www.openfoodsystem.fr

http://cordis.europa.eu/project/rcn/198017_en.html

http://www.cistern.nl/index.php/consortium

http://www-iem.u-bourgogne.fr/MASTER/MSCAESE/homepage_128.htm

Acknowledgments

I would like to express my gratitude to Pierre Gouton with whom I have collaborated for more than 10 years now. Pierre has humanistic beliefs and implements them at work better than most of us. It is also thanks to him that I have been able to carry out my research with much freedom.

Education

• PhD, Color imaging science (2009), from Université de Bourgogne, France, in collaboration with the Gjøvik University College, Norway.

• MSc, Optics, Image and Vision, with major in Image, Vision and Signal processing (2006), from Université Jean Monnet, Saint-Etienne, France.

• BSc, Applied Physics (2004), from Université Jean Monnet.

Scientic History

• Post doctoral fellow, since September 2016 at NTNU-Gjøvik.

Interests: Measuring and understanding the visual appearance of complex 3D translucent objects.

Projects: MUVApp.

• Maître de conférences, since September 2010 at Université de Bourgogne, Franche-Comté.

Interests: Multispectral lter array for color and multispectral acquisition (lter design, optimization, demosaicing, color constancy). I recently get involved more into illumination estimation from uncalibrated multispectral images since I do believe this is a critical leverage to take these cameras out of the labs.

Major projects: Open Food System (PSPC), EXIST(H2020), CISTERN (CATRENE). • Dr Pierre-Jean Lapray is now Associate Professor (Maître de Conférences) at Université de Haute-Alsace.

• Dr Keivan Ansari is back to Iran where he is Assistant Professor at the Institute for Colour Science and Technology in Tehran. • Dr Jessica El Khoury is now ATER at Université de Bourgogne in Auxerre. She opened her expertise toward multi-angle imaging and surface inspection.

• Haris Ahmad Khan is in the process of writing his thesis. He has got oered several post doctoral positions already, so I have no doubt that he will do very well after.

Other

• I am occasionally Master thesis external examinator at HIG/NTNU-Gjøvik and at EPFL.

• I supervise every year Master and Bachelor students on smaller projects.

• I was invited to a jury for a PhD thesis defense (Hasan SHEIKH FARIDUL, Université Jean Monnet, the 06/01/2014) I wrote the proposal about multispectral imaging for the Le2i and would have managed the projects if I had not to quit to the détachement. Pierre Gouton followed up after my leave.

OFS

Open Food System aims at dening the kitchen of tomorrow by the mean of connected and instrumented cooking devices. I managed this project, funded by the ministry of industries, for the Le2i. This project started in January 2013 and ended in July 2016.

I wrote the proposal and managed the project for the Le2i. We wrote the proposal together with Benjamin, who is the project manager. At rst the partnership was with the Le2i, but the project followed me to Norway.