Sofiane Mihoubi

Olivier Losson
email: olivier.losson@univ-lille.fr

Benjamin Mathon

AND Ludovic Macaire

Spatio-spectral binary patterns based on multispectral filter arrays for texture classification

To discriminate gray-level texture images, spatial texture descriptors can be extracted using the local binary pattern operator. This operator has been extended to color images at the expense of increased memory and computation requirements. Some authors propose to compute texture descriptors directly from raw images provided through a Bayer color filter array, which both avoids the demosaicing step and reduces the descriptor size. Recently, multispectral snapshot cameras have emerged to sample more than three wavelength bands using a multispectral filter array. Such cameras provide a raw image in which a single spectral channel value is available at each pixel. In this paper we design a local binary pattern operator that jointly extracts the spatial and spectral texture information directly from a raw image. Extensive experiments on a large dataset show that the proposed descriptor has both reduced computation cost and high discriminative power with regard to classical LBP descriptors applied to demosaiced images.

INTRODUCTION

Among multispectral imaging technologies, "linescan" devices acquire frames of narrow spatial slices for several spectral bands then, once the whole scene has been scanned, form the multispectral image by stacking the acquired frames. "Snapshot" devices can oppositely acquire the multispectral image from a single shot. Multi-sensor snapshot systems straight form a fully-defined multispectral image thanks to dichroic beam splitters that selectively redirect the incoming light by wavelength onto the sensors. Because these systems are expensive and are sensitive to a limited number of bands, single-sensor snapshot systems have been recently developed [START_REF] Hagen | Review of snapshot spectral imaging technologies[END_REF][START_REF] Martínez | Combining transverse field detectors and color filter arrays to improve multispectral imaging systems[END_REF]. Most of them use a multispectral filter array (MSFA) laid over the sensor that spectrally samples the incident light, like the widely-used Bayer color filter array (CFA) in color imaging. The MSFA is defined by a basic periodic pattern in which each filter is sensitive to a narrow spectral band. Each pixel of the resulting raw image is then characterized by one single band according to the MSFA. Such technology achieves a compromise between spatial and spectral samplings (see Sec. A). The fully-defined multispectral image is estimated by a demosaicing process (see Sec. B). This process may be greedy and is prone to generate spatio-spectral estimation artifacts. We therefore propose to avoid it for texture classification and design a texture descriptor that is directly computed from the raw image (see Sec. C).

A. Multispectral filter arrays

Like the Bayer CFA in which the green (G) band is overrepresented with regard to red (R) and blue (B) bands, some MSFA patterns in the literature include one or several dominant bands [START_REF] Lapray | Multispectral Filter Arrays: Recent Advances and Practical Implementation[END_REF]. For instance, Monno et al. [START_REF] Monno | Multispectral demosaicking using adaptive kernel upsampling[END_REF] propose the 4 × 4 basic pattern shown in Fig. 1a that samples 5 bands in the visible (Vis) domain. The dominant G band characterizes half of the pixels and the other four bands evenly characterize the other half. This is a special case of the generic pattern generated by a binary tree [START_REF] Miao | Binary Tree-based Generic Demosaicking Algorithm for Multispectral Filter Arrays[END_REF], where the prior probability of the G band is 1 2 and that of the other bands is 1 8 . Thomas et al. [START_REF] Thomas | Spectral Characterization of a Prototype SFA Camera for Joint Visible and NIR Acquisition[END_REF] propose the basic redundant pattern shown in Fig. 1b that samples 8 bands in the visible and near infrared (NIR) domains. The two MSFAs based on the above basic patterns, called Vis5 and VisNIR8 in the following, own the desirable properties of spectral consistency and spatial uniformity [START_REF] Miao | Binary Tree-based Generic Demosaicking Algorithm for Multispectral Filter Arrays[END_REF]. An MSFA is spectrally consistent if the same bands are sampled the same number of times in the neighborhood of all filters associated to any given band. Spatial uniformity requires that an MSFA spatially samples each band as evenly as possible. Both requirements are related to the demosaicing process that is applied to the raw image. Indeed, demosaicing independently scans all the pixels associated to a given band and considers pixels in their neighborhood. This neighborhood layout should then be same whatever the pixel considered in the raw image. [START_REF] Monno | Multispectral demosaicking using adaptive kernel upsampling[END_REF], (b) Vis-NIR8 [START_REF] Thomas | Spectral Characterization of a Prototype SFA Camera for Joint Visible and NIR Acquisition[END_REF], (c) IMEC16 and (d) IMEC25 [START_REF] Geelen | A compact snapshot multispectral imager with a monolithically integrated per-pixel filter mosaic[END_REF]. Numbers are band indices. Vis5 band labels (Or=Orange, Cy=Cyan) are those of [START_REF] Monno | Multispectral demosaicking using adaptive kernel upsampling[END_REF] but could also be replaced by indices. Dominant G band is marked as white.

Increasing the number of bands to enhance spectral resolution is a goal of multispectral imaging. Some MSFAs are then defined by a basic pattern with no dominant band although this conflicts with a dense spatial sampling. The two MSFAs whose square basic patterns are shown in Figs. 1c and 1d are manufactured by IMEC [START_REF] Geelen | A compact snapshot multispectral imager with a monolithically integrated per-pixel filter mosaic[END_REF]. They are embedded in a few off-the-shelf MSFA-based devices available on the market, like XIMEA xiSpec and IMEC "snapshot mosaic" multispectral cameras, with applications in medical imaging [START_REF] Pichette | Intraoperative video-rate hemodynamic response assessment in human cortex using snapshot hyperspectral optical imaging[END_REF] or terrain classification [START_REF] Winkens | Robust features for snapshot hyperspectral terrain-classification[END_REF]. The 4 × 4 basic pattern samples 16 bands centered at wavelengths λ 1 = 469 nm, . . . , λ 16 = 633 nm (visible domain) and the 5 × 5 one samples 25 bands centered at λ 1 = 678 nm, . . . , λ 25 = 960 nm (visible and NIR domains). Note that the band centers are not equally spaced and that they are not ascending in the classical pixel readout order (see Figs. 1c and1d). The MSFAs defined by these patterns (or the corresponding cameras) are shortly called IMEC16 and IMEC25 in the following.

B. MSFA demosaicing

In the raw image acquired through an MSFA that samples K bands, a single band value is available at each pixel. A procedure called demosaicing must be applied to this raw image in order to obtain a multispectral image with K fully-defined channels, among which K -1 ones are estimated at each pixel.

The simplest way to demosaic the raw image is to use a weighted bilinear (WB) interpolation filter that estimates each missing band value at a given pixel by using the available values of the same band at its neighbors. Each neighbor is associated with a weight that depends on its spatial distance to the considered pixel [START_REF] Brauers | A Color Filter Array Based Multispectral Camera[END_REF]. This method only exploits spatial correlation within each channel to estimate the missing values.

To improve the demosaicing performances, some authors also use spectral (inter-channel) correlation. Brauers and Aach [START_REF] Brauers | A Color Filter Array Based Multispectral Camera[END_REF] compute the channel differences and Mizutani et al. [START_REF] Mizutani | Multispectral demosaicking algorithm based on inter-channel correlation[END_REF] iterate this scheme a number of times given by the spectral distance between band centers of the considered pair of bands. Wang et al. [START_REF] Wang | Discrete wavelet transform based multispectral filter array demosaicking[END_REF] extend the CFA demosaicing method based on the discrete wavelet transform to MSFA demosaicing. Mihoubi et al. [START_REF] Mihoubi | Multispectral demosaicing using pseudo-panchromatic image[END_REF] estimate the pseudo-panchromatic image (PPI) from the raw image and propose a demosaicing scheme based on PPI differences (PPID).

When the MSFA includes a dominant band (as does Vis5 in Fig. 1a), a standard approach is to pre-estimate the dominant channel, then to rely on it to estimate the other missing channels. The binary tree-based edge-sensing method of Miao et al. [START_REF] Miao | Binary Tree-based Generic Demosaicking Algorithm for Multispectral Filter Arrays[END_REF] estimates missing values from the edge information of the preestimated dominant channel. However, this method is only applicable when the prior probability of each band is 1 2 n , n ∈ N, which notably excludes IMEC25 MSFA. Monno et al. propose three demosaicing methods that exploit the dominant G band of their Vis5 MSFA. The original algorithm [START_REF] Monno | Multispectral demosaicking using adaptive kernel upsampling[END_REF] that uses an adaptive kernel up-sampling is improved by guided filtering [START_REF] Monno | Multispectral demosaicking using guided filter[END_REF], and further on by residual interpolation [START_REF] Monno | Multispectral Demosaicking with Novel Guide Image Generation and Residual Interpolation[END_REF].

C. Our contribution to texture classification

To classify multispectral texture images acquired by singlesensor snapshot cameras, the classical approach is to demosaic the raw images, extract texture features from the estimated images, then compare features that are computed from images thanks to a similarity measure.

In this paper, we focus on texture features based on local binary patterns (LBPs). The many variants of LBP operators have indeed proved to be very efficient for a wide variety of applications [START_REF] Pietikäinen | Computer vision using local binary patterns[END_REF]. LBP-based texture classification has first been performed on gray-level images since the original operator only uses the spatial information of texture [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF]. Later, Palm [START_REF] Palm | Color texture classification by integrative Co-occurrence matrices[END_REF] has shown that classification based on a color analysis outperforms that based on the spatial information only. Texture feature extraction is then extended to the color domain by taking both spatial and spectral textural information into account.

Recent advances in multispectral imaging lead us to extend the color texture features to the multispectral domain (see Sec. 2). But the computational cost significantly increases with the number of channels due to demosaicing and feature extraction. Thus we propose a new computationally-efficient LBP-based descriptor that is directly computed from raw images, which allows us to avoid the demosaicing step (see Sec. 3). Extensive experiments on a large dataset of multispectral texture images prove the relevance of our approach (see Sec. 4).

SPECTRAL TEXTURE FEATURES BASED ON LOCAL BINARY PATTERNS

Below we formulate several LBP-based texture features for any fully-defined K-channel image, that we generically denote as I = {I k } K k=1 for simplicity whatever the value of K and even it has been estimated by demosaicing. For a given pixel p, we consider the neighborhood N p defined by its support N P,d made of P pixels at spatial uniform distance d from p. We now detail how the LBP operator compares the pixel values in the neighborhood N p with respect to the value of p.

A. Marginal LBPs

The marginal LBP operator (see Fig. 2) is defined for each band (index) k ∈ {1, ..., K} at each pixel p as

LBP k [I](p) = ∑ q∈N p s I k q , I k p • 2 (q) , (1)
where I k p is the value of channel I k at p, (q) ∈ {0, ..., P -1} is the index of each neighboring pixel q in N p , and s(•) is the unit

, I k p q∈Np Weights 2 ε(q) q∈Np LBP k [I](p) = 1 + 4 + 8 + 64 = 77 Neighborhood of p in I k s I k q , I k p • 2 ε(q) q∈Np Eq. ( 1
)
Eq. ( 2)

Fig. 2. Marginal LBP operator applied to a pixel p of channel I k . step function:

s(α, β) = 1 if α ≥ β, 0 otherwise. (2)
Each channel I k is characterized by the 2 P -bin un-normalized histogram of its LBP values. The multispectral texture image I is then described by the concatenation of the K histograms of {LBP k [I]} K k=1 . This feature, whose size is K • 2 P , represents the spatial interaction between neighboring pixels within each channel independently.

Note that we only consider the basic LBP operator in this paper even though many LBP variants have been described in the literature [START_REF] Pietikäinen | Computer vision using local binary patterns[END_REF]. Also note that the definition of Eq. ( 1) ignores border effects for readability sake and that only those pixels at which N p is fully enclosed in the image are actually taken into account to compute the LBP histogram.

B. Moment LBPs

Mirhashemi [START_REF] Mirhashemi | Introducing spectral moment features in analyzing the SpecTex hyperspectral texture database[END_REF] proposes an LBP-based spectral feature using mathematical moments to characterize the reflectance spectrum shape. The LBP operator of Eq. ( 1) is no longer applied to pixel values but to moment values of the pixel spectral signatures.

Different moments can be extracted from the reflectance {R p (λ k )} K k=1 sampled over K bands at each pixel p. Raw and central type-I moments of order n ∈ N are defined as

M n (p) = K ∑ k=1 λ k n R p (λ k ) and µ n (p) = K ∑ k=1 λ k - M 1 (p) M 0 (p) n R p (λ k ). (3)
Type-II moments are estimated moments of the probability density function from which reflectance values are sampled. Raw and central type-II moments are expressed as

Mn (p) = 1 K K ∑ k=1 R p (λ k ) n and μn (p) = 1 K K ∑ k=1 R p (λ k ) -M1 (p) M0 (p) n . (4)
Alternatively, these moments can be computed from the reflectance normalized by its L 1 -norm at each pixel p:

r p (λ k ) = R p (λ k ) ∑ K i=1 R p (λ i )
. We then denote type-I and type-II raw moments as m n (p) and mn (p).

Mirhashemi assesses the texture classification performance of all the possible moment-based features (namely the 38 moment LBP histograms obtained for n = 1..6), either considered alone or concatenated in 2-or 3-feature combinations. The most powerful combinations use three features based on the following moments: m 1 (p) or m1 (p), M 1 (p) or M1 (p), and µ 3 (p), µ 5 (p), μ3 (p), or μ5 (p) [START_REF] Mirhashemi | Introducing spectral moment features in analyzing the SpecTex hyperspectral texture database[END_REF]. The texture feature is then a concatenated histogram with 3 • 2 P bins.

C. Map-based LBPs

Dubey et al. [START_REF] Dubey | Multichannel Decoded Local Binary Patterns for Content-Based Image Retrieval[END_REF] propose two kinds of LBP operators that can theoretically be applied to any K-channel image. These operators use the spectral information in the encoding scheme by testing the sum of the marginal comparison patterns between each pixel p and its neighbors over all channels. The adder-based LBPs {maLBP m } K m=0 are defined as

maLBP m [I](p) = ∑ q∈N p 2 (q) if ∑ K k=1 s(I k q , I k p ) = m, 0 otherwise. (5)
The decoder-based LBPs {mdLBP n } 2 K -1 n=0 are defined as

mdLBP n [I](p) = ∑ q∈N p 2 (q) if ∑ K k=1 s(I k q , I k p ) • 2 (K-k) = n, 0 otherwise. (6)
The concatenation of the operator's histograms provides the final feature of size (K + 1) • 2 P (maLBPs) or 2 K • 2 P (mdLBPs).

D. Luminance-spectral LBPs

By analogy with the luminance-chrominance model for a color image, a multispectral image can be represented as both a panchromatic channel and the joint information computed from two or more channels. The PPI that carries the spatial information of the luminance is computed as the average value over all channels at each pixel p [START_REF] Mihoubi | Multispectral demosaicing using pseudo-panchromatic image[END_REF]:

Īp = 1 K K ∑ k=1 I k p . (7)
To form the final feature, the histogram of the output of the LBP operator applied to Ī is concatenated with a histogram based on the spectral content according to one of the following propositions that we extend here to the multispectral domain:

• Cusano et al. [START_REF] Cusano | Combining local binary patterns and local color contrast for texture classification under varying illumination[END_REF] define the local color contrast (LCC) operator that depends on the angle between the value of a pixel p and the average value Īp = 1 P ∑ q∈N p I q of its neighbors in the spectral domain:

LCC[I](p) = cos -1 I p , Īp ||I p || • || Īp || , (8)
where • Lee et al. [START_REF] Lee | Local Color Vector Binary Patterns From Multichannel Face Images for Face Recognition[END_REF] consider I in a K-dimensional space and compute spectral angular patterns between bands at each pixel. Specifically, for each pair of bands (k, l) ∈ {1, ..., K} 2 , k = l, the authors apply the LBP operator to the image Ĩk,l defined at each pixel p as the angle between the axis of the band k and the projection of I p onto the plane associated with bands k and l:

Ĩk,l p = tan -1 I k p I l p + η , ( 9
)
where η is a small-valued constant to avoid division by zero. The histogram of

LBP[ Ī] is concatenated to the K(K - 1) histograms of {LBP[ Ĩk,l ]} K k,l=1,k =l to provide the final feature of size (1 + K(K -1)) • 2 P .

E. Opponent band LBPs

To fully take spectral correlation into account, Mäenpää et al. [START_REF] Mäenpää | Separating color and pattern information for color texture discrimination[END_REF] apply the opponent color LBP (OCLBP) operator to each pair of channels of a color image. This operator can be directly generalized as the opponent band LBP (OBLBP) applied to each pair of channels (I k , I l ), (k, l) ∈ {1, ..., K} 2 , of a multispectral image:

OBLBP k,l [I](p) = ∑ q∈N p s I l q , I k p • 2 (q) . (10)
Bianconi et al. [START_REF] Bianconi | Improved opponent color local binary patterns: an effective local image descriptor for color texture classification[END_REF] similarly consider both intra-and interchannel information but with a different thresholding scheme. Their improved OBLBP (IOBLBP) operator uses a local average value rather than the sole central pixel value as threshold:

IOBLBP k,l [I](p) = ∑ q∈{p}∪N p s I l q , Īk p • 2 (q) , (11)
where Īk p = 1 P+1 ∑ r∈{p}∪N p I k r and (p) = P. In both cases, the texture feature is the concatenation of the K 2 2 P -bin histograms of {(I)OBLBP k,l [I]} K k,l=1 .

MSFA TEXTURE FEATURE BASED ON LOCAL BI-NARY PATTERNS

We intend to design an LBP-like operator to characterize multispectral texture images directly from the raw image acquired by snapshot cameras. A similar approach was proposed by Losson and Macaire [START_REF] Losson | CFA local binary patterns for fast illuminantinvariant color texture classification[END_REF] for color texture representation from raw CFA images. Rather than a straightforward extension that would neglect spectral correlation, we here propose a new operator dedicated to raw MSFA images and inspired by OBLBPs.

We first present the raw image in details in this section, including the specific neighborhoods defined by the MSFA. Then we describe our operator based on these MSFA neighborhoods, and we explain how it is related to OBLBPs.

A. MSFA neighborhoods

An MSFA associates a single spectral band with each pixel. It can be defined as a function MSFA: S → {1, . . . , K} over the set S of all pixels. Let S k = {p ∈ S, MSFA(p) = k} be the pixel subset where the MSFA samples the band k, such that S = K k=1 S k . In other words, S k is the subset of pixels that are associated with the band k. Figure 3 shows the example of the IMEC16 MSFA and one among its 16 pixel subsets.

For a given pixel p ∈ S k , k ∈ {1, . . . , K}, let B k = {l ∈ {1, . . . , K}, MSFA(q) = l} q∈N p be the set of bands that are associated with the neighboring pixels in N p according to the MSFA. Note that N p is always composed of pixels with the same associated bands whatever the location of p. Moreover, we assume that any neighbor q ∈ N p is always associated with the same band for a given relative position with respect to p in the MSFA pattern. A necessary but not sufficient condition for this assumption to be fulfilled is spectral consistency. Then, the neighborhood of p ∈ S k can be decomposed into

N p = l∈B k N k,l p , (12)
where N k,l p = N p ∩ S l is the MSFA-based neighborhood made of the neighboring pixels of p that belong to S l . Let us notice that N k,l p = ∅ ⇐⇒ l ∈ B k and stress out that B k and N k,l p both depend on N P,d and on the basic MSFA pattern.

For illustration purposes, let us consider the IMEC16 and VisNIR8 MSFAs and focus on the 3 × 3 neighborhood defined by the support N 8,1 as shown in Fig. 4

B. MSFA-based LBPs

A snapshot multispectral camera provides a raw image I raw in which a single band is associated with each pixel according to the MSFA. Then, I raw can be seen as a spectrally-sampled version of the reference fully-defined image I = {I k } K k=1 (that is unavailable in practice) according to the MSFA: To design a texture feature dedicated to the raw image, let us first consider applying the basic LBP operator of Eq. ( 1) directly to I raw considered as a gray-level image:

MLBP [I raw ] (p) = ∑ q∈N p s(I raw q , I raw p ) • 2 (q) . ( 14
)
The LBP operator is here renamed as MSFA-based LBP (MLBP) to make clear the key difference introduced by its application to raw and its dependency upon the considered MSFA. Unlike Eq. (1), Eq. ( 14

C. Relation between MSFA-based and opponent band LBPs

To show that the MSFA-based LBP defined by Eq. ( 14) bears an analogy to OBLBP (see Eq. ( 10)), let us consider its output as the direct sum of the sparse outputs of the same operator restrictively applied to each pixel subset

S k : im MLBP [I raw ] = K k=1 im MLBP S k [I raw ] , (16)
where im{•} is a function output. According to the definition of S k , we have I raw p = I k p for each pixel p ∈ S k . From Eq. ( 14) and the decomposition of the neighborhood N p according to Eq. ( 12), we can then express MLBP S k from {I k } K k=1 as

MLBP S k [I raw ] (p) = ∑ l∈B k ∑ q∈N k,l p s(I l q , I k p ) • 2 (q) . (17)

EXPERIMENTS A. Experimental settings

A.1. Texture dataset and classification scheme

In order to perform texture classification, we use HyTexiLa database that contains the reflectance images of 112 textured materials (see Fig. 6) and is currently the most suitable database in the VisNIR domain [START_REF] Khan | HyTexiLa: High Resolution Visible and Near Infrared Hyperspectral Texture Images[END_REF]. We consider three CIE standard illuminants (E, D65, and A) to simulate three fully-defined images from the reflectance of each texture. Finally, we sample these images according to several MSFAs to simulate the raw images that would be acquired by various snapshot multispectral cameras. We consider the four MSFAs of Fig. 1 since they are either related to research works with detailed publications (Vis5, Vis-NIR8) or available in consumer cameras (IMEC). The spectral sensitivity functions (SSFs) of Vis5 and VisNIR8 can be found in the respective papers [START_REF] Monno | A Practical One-Shot Multispectral Imaging System Using a Single Image Sensor[END_REF] and [START_REF] Thomas | Spectral Characterization of a Prototype SFA Camera for Joint Visible and NIR Acquisition[END_REF]. Figure 5 shows the normal-ized SSFs of IMEC16 and IMEC25 cameras [START_REF] Geelen | A compact snapshot multispectral imager with a monolithically integrated per-pixel filter mosaic[END_REF] that respectively sample the visible light around λ 1 = 469 nm, . . . , λ 16 = 633 nm and NIR light around λ 1 = 678 nm, . . . , λ 25 = 960 nm. This figure shows that the SSFs of bands with close centers may overlap to a large extent. Note that the SSFs include the optics and sensor sensitivities and the optical band-pass filter that removes spectral artifacts.

HyTexiLa reflectances are defined on 186 bands whose centers range from λ min = 405.37 nm to λ max = 995.83 nm with a step of 3.19 nm. From these data and the three illuminations provided by Thomas et al. [START_REF] Thomas | Spectral Characterization of a Prototype SFA Camera for Joint Visible and NIR Acquisition[END_REF] (that extend A and E illuminants to the NIR domain and simulate D65), we compute the images of illuminated textures as [START_REF] Mihoubi | Multispectral demosaicing using pseudo-panchromatic image[END_REF]

I k p = Q λ max ∑ λ=λ min E(λ)R p (λ)SSF k (λ) . (18)
In this equation, Q(•) is the radiance quantization function on 8 bits, and E(λ), R p (λ), and SSF k (λ) are the illumination, the reflectance of the surface elements captured at pixel p, and the camera-specific SSF of the band k, respectively. The values of these discrete functions of λ are obtained at 1-nm resolution by linear interpolation of the original values. To prevent saturation in the simulated images, illuminations and SSFs are scaled so that max λ E(λ) = 255 and max k ∑ λ SSF k (λ) = 1, and reflectance values above 1 (that correspond to specular surfaces) are clipped to 1. This normalization practically corresponds to setting the integration time of the camera as the limit before saturation when a white patch is observed. This has an effect on the dynamic range of channels [START_REF] Lapray | Energy balance in Spectral Filter Array camera design[END_REF].

For each of the considered four cameras and three illuminations, these simulations provide 112 raw images of size 1024 × 1024 pixels. Each image is then split (leaving out the last four columns and rows) into 25 sub-images of size 204 × 204 pixels, among which 12 are randomly picked for training and the 13 others for testing.

In order to determine the most discriminative texture feature, we retain the 1-nearest neighbor classifier coupled with the similarity measure based on intersection between histograms [START_REF] Swain | Color indexing[END_REF] since this classification scheme requires no parameter.

A.2. MSFA neighborhoods

As explained in Sec. 3A, the neighbors of any pixel p are associated with different bands according to the MSFA. It is thus impossible to consider interpolated values in a circular neighborhood of p as is usually done for LBP-like operators. To avoid interpolation, we therefore consider the uniform spatial distance (hence square neighborhoods) rather than the Euclidean one. Moreover, LBP operators classically use neighborhoods with P = 8, 16, or 24 pixels. But P = 16 with d = 3 does not match the image lattice and requires interpolation, and 24 would yield extremely large features. We therefore set P = 8 and consider the three supports N 8,d with uniform distance d ∈ {1, 2, 3} as shown in Fig. 7. with IMEC25 due to the large 5 × 5 basic pattern of this MSFA. Note that |B k | reflects the degree to which spectral correlation is taken into account by an MSFA neighborhood.

G 1 G 1 G 1 G 1 G 1 G 1 G 1 G 1 G 2 G 2 G 2 G 2 G 2 G 2 G 2 G 2 G 3 G 3 G 3 G 3 G 3 G 3 G 3 G 3 G 3 G 3 G 4 G 4 G 4 G 4 G 4 G 4 G 4 G 4 G 4 G 4 R R R R R R R R R R
The basic pattern of the Vis5 MSFA (see Fig. 1a) is particular because of its single dominant G band. Unlike in the other considered MSFAs, |B k | in Vis5 depends on d but also on p (i.e., on k) (see Tab.1). Considering d = 1 for instance, the neighbors of a pixel p 1 ∈ S G belong to all the bands (hence |B G | = 5) while those of a pixel p 2 ∈ S Cy belong to B Cy = {R, G, Or}. Moreover, Vis5 contradicts our assumption that a neighbor of p ∈ S k is always associated with the same band for a given relative position whatever the location of p. Indeed, for two pixels associated with the G band, vertical neighbors may either be associated with R and Or or with B and Cy. To fulfill our assumption and compute MLBP with Vis5, we therefore split S G into four pixel subsets {S G i } 4 i=1 as shown in Fig. 7a. The information of {MLBP S

G i [I raw ]} 4 i=1 is then merged into a single histogram h G [MLBP [I raw ]] for the G band.

A.3. Feature extraction

Table 2 summarizes the sizes of the texture features described in Sec. 2 as the size of each histogram (that depends on P) and the number of histograms (that depends on K). Setting P = 8 makes the histogram size to be 256 but the mdLBP operator provides a prohibitively large number of histograms when K ≥ 16. All approaches but mdLBP are hence tested against our MSFA-based LBP in the experiments. Besides, among the 16 moment combinations from the format {m 1 | m1 }{M 1 | M1 }{µ 3|5 | μ3|5 } (see Sec. 2B), we only retain m 1 M 1 µ 3 whose LBP histogram provides the best classification result on average over all the experiments.

The number of histograms may impact classification accuracy and computational burden. The approaches can then be divided into three groups, depending on whether this number is constant (Cusano and Moment LBPs), proportional to K (Marginal LBPs, maLBP, and MLBP), or to K 2 (OBLBP, IOBLBP, and Lee LBPs).

Computation cost also deserves some attention as an implementation-independent indication of the required processing time. The last two columns of Tab. 2 show this cost as the number of elementary operations per pixel required to compute a feature. This estimation includes all arithmetic operations at the same cost of 1 but excludes array indexing.

All features but ours first require to estimate a fully-defined multispectral image by demosaicing. In our experiments, we only consider WB [START_REF] Brauers | A Color Filter Array Based Multispectral Camera[END_REF] that is both the most simple and generic method (see Sec. 1B) and PPID [START_REF] Mihoubi | Multispectral demosaicing using pseudo-panchromatic image[END_REF] that provides the best demosaicing results in most cases. We have adapted PPID to the Vis5 MSFA and we retain it because it globally yields better classification results than the dedicated guided filtering method provided by [START_REF] Monno | A Practical One-Shot Multispectral Imaging System Using a Single Image Sensor[END_REF]. Since demosaicing is neither our main concern nor the greedier feature computation step, we minimally evaluate its number of operations as that of the weighted average of two values required by WB to estimate each missing value at a pixel, namely 4(K -1).

The feature computation costs given in the last column of Tab. 2 result from the equation(s) of each feature recalled in the second column. As previously stated, the computation cost of mdLBP is prohibitive. Our MLBP-based feature requires 24 operations per pixel. In contrast, the cost of Marginal LBPs, maLBP, and Cusano LBPs grows with K, and that of other approaches with K 2 . By considering both the feature size and computation cost, Lee LBPs, OBLBP and IOBLBP are the most greedy features. MLBP is the most efficient and is represented by the same number of histograms as Marginal LBPs and maLBP. This should be kept in mind while analyzing the classification results.

B. Experimental results

B.1. Accuracy vs. computation cost

We first propose a study to highlight the above remark about feature computation costs. Let us consider the case d = 1 and the D65 illuminant, and assess the classification accuracy provided by each feature with regard to its cost. For all approaches except MLBP, WB is chosen to demosaic the raw image.

Figure 8 separately shows the results for the four considered MSFAs. OBLBP globally outperforms other features but at a very high computation cost. MLBP provides only slightly lower results than OBLBP for Vis5 and VisNIR8 MSFAs and similar results for IMEC MSFAs, at about a K 2 times smaller cost. Considering features with comparable costs, Lee LBPs and IOBLBP perform worse than OBLBP. Moment LBPs generally provide fair results with regard to the other three features with moderate costs (marginal LBPs, maLBP, and Cusano LBPs). MLBP clearly outperforms the latter four features in all cases with the benefit of reduced computation requirements.

B.2. Classification results and discussion

We now extensively assess the performances of our MLBP-based feature with respect to those of other features in various experimental conditions. Table 3 shows the classification accuracies provided by the different features for each of the four MSFAs using three illuminants (E, D65, and A), three distance values (d ∈ {1, 2, 3}), and two demosaicing methods (WB and PPID). For each combination of experimental settings (illuminant, distance, and demosaicing method), the better classification accuracies than that of our descriptor are highlighted as bold.

Let us first study the behavior of our MLBP-based feature with respect to the different settings. The neighborhood distance d slightly influences MLBP performances for IMEC25 MSFA (see Tab. 3d) whereas d = 2 provides clearly lower performances than d ∈ {1, 3} for IMEC16 (see Tab. 3c). This was expected since d determines the number |B k | of available bands in a pixel neighborhood, that does not depend on d for IMEC25 but is reduced from 8 to 3 bands when d = 2 for IMEC16 (see Tab. 1). Regarding Vis5 and VisNIR8, there is no systematic performance loss for d = 2 though |B k | is reduced. This is because the poor information about spectral (inter-channel) correlation in these cases is completed by taking spatial (intra-channel) correlation into account. When d = 2 indeed, k ∈ B k (see Figs. 7a and7b), which means that MLBP takes both intra-and inter-channel correlation into account. No such case occurs with IMEC16 and IMEC25 (see Figs. 7c and7d) with which MLBP only takes interchannel correlation into account because k ∈ B k whatever the value of d. Two other outstanding results are obtained with Vis5 MSFA and illuminant A when d ∈ {1, 3}. We explain these 7a). Therefore, due to the particular band arrangement in Vis5 MSFA, MLBP is less discriminative and performances are notably reduced when d ∈ {1, 3} with A illuminant.

Let us now compare the performances reached by our descriptor with those of other approaches. Table 3 shows that, except with Vis5 MSFA images simulated under A illuminant, our MLBP-based feature always outperforms approaches with either smaller (Cusano and Moment LBPs) or similar-size features (Marginal LBPs and maLBP). Moreover, our lightweight approach obtains close results to greedy ones (Lee LBP, OBLBP and IOLBP), especially with IMEC MSFAs, and even performs better than them in 95 out of the 216 tested cases. The best accuracy reached by MLBP is 97.32% (with IMEC16 MSFA under D65 illuminant using d = 1) while the best descriptor (OBLBP) reaches 97.60% (with the same settings and PPID demosaicing).

CONCLUSION

In this paper, we introduce a conceptually simple and highlydiscriminative LBP-based feature for multispectral raw images. In addition to its algorithmic simplicity, our operator is directly applied to raw images, which allows it to avoid the demosaicing step and keeps its computational cost low. We perform extensive experiments of texture classification on simulated multispectral images with four well-referenced MSFAs. The results show that the proposed approach outperforms existing ones using features of similar sizes, and provides comparable results to that of features with large size and high computational cost.

Future works will further study how our feature embeds spatial and spectral correlations according to the MSFA and neighborhood parameters. Since MLBP is a small-size feature, there is room for additional correlation information that could still improve its classification results. For instance, it could be made more robust to the neighborhood distance by concatenating several MLBP histograms computed with different distances. Another approach to improve how our feature accounts for spatial correlation would be to use a demosaiced dominant channel. Other investigations could at last focus on the spectral distance among the considered neighbors. Marginal LBPs [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] Moment LBPs [START_REF] Mirhashemi | Introducing spectral moment features in analyzing the SpecTex hyperspectral texture database[END_REF] maLBP [START_REF] Dubey | Multichannel Decoded Local Binary Patterns for Content-Based Image Retrieval[END_REF] Cusano LBPs [START_REF] Cusano | Combining local binary patterns and local color contrast for texture classification under varying illumination[END_REF] Lee LBPs [START_REF] Lee | Local Color Vector Binary Patterns From Multichannel Face Images for Face Recognition[END_REF] OBLBP [START_REF] Mäenpää | Separating color and pattern information for color texture discrimination[END_REF] IOBLBP [START_REF] Bianconi | Improved opponent color local binary patterns: an effective local image descriptor for color texture classification[END_REF] MLBP (ours)

A d = 1 d = 2 d = 3 d = 1 d = 2 d = 3 d = 1 d = 2 d = 3 WB PPID WB
Approach E D65 A d = 1 d = 2 d = 3 d = 1 d = 2 d = 3 d = 1 d = 2 d = 3 WB PPID WB
Computation cost (operations/pixel) Classification accuracy (%) Marginal LBPs [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] Moment LBPs [START_REF] Mirhashemi | Introducing spectral moment features in analyzing the SpecTex hyperspectral texture database[END_REF] maLBP [START_REF] Dubey | Multichannel Decoded Local Binary Patterns for Content-Based Image Retrieval[END_REF] Cusano LBPs [START_REF] Cusano | Combining local binary patterns and local color contrast for texture classification under varying illumination[END_REF] Lee LBPs [START_REF] Lee | Local Color Vector Binary Patterns From Multichannel Face Images for Face Recognition[END_REF] OBLBP [START_REF] Mäenpää | Separating color and pattern information for color texture discrimination[END_REF] IOBLBP [START_REF] Bianconi | Improved opponent color local binary patterns: an effective local image descriptor for color texture classification[END_REF] MLBP (ours)

Computation cost (operations/pixel) Classification accuracy (%) Marginal LBPs [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] Moment LBPs [START_REF] Mirhashemi | Introducing spectral moment features in analyzing the SpecTex hyperspectral texture database[END_REF] maLBP [START_REF] Dubey | Multichannel Decoded Local Binary Patterns for Content-Based Image Retrieval[END_REF] Cusano LBPs [START_REF] Cusano | Combining local binary patterns and local color contrast for texture classification under varying illumination[END_REF] Lee LBPs [START_REF] Lee | Local Color Vector Binary Patterns From Multichannel Face Images for Face Recognition[END_REF] OBLBP [START_REF] Mäenpää | Separating color and pattern information for color texture discrimination[END_REF] IOBLBP [START_REF] Bianconi | Improved opponent color local binary patterns: an effective local image descriptor for color texture classification[END_REF] MLBP (ours)

Computation cost (operations/pixel) Classification accuracy (%) Marginal LBPs [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] Moment LBPs [START_REF] Mirhashemi | Introducing spectral moment features in analyzing the SpecTex hyperspectral texture database[END_REF] maLBP [START_REF] Dubey | Multichannel Decoded Local Binary Patterns for Content-Based Image Retrieval[END_REF] Cusano LBPs [START_REF] Cusano | Combining local binary patterns and local color contrast for texture classification under varying illumination[END_REF] Lee LBPs [START_REF] Lee | Local Color Vector Binary Patterns From Multichannel Face Images for Face Recognition[END_REF] OBLBP [START_REF] Mäenpää | Separating color and pattern information for color texture discrimination[END_REF] IOBLBP [START_REF] Bianconi | Improved opponent color local binary patterns: an effective local image descriptor for color texture classification[END_REF] MLBP (ours)

Computation cost (operations/pixel) Classification accuracy (%)

Fig. 1 .

 1 Fig. 1. Basic patterns of different MSFAs: (a) Vis5[START_REF] Monno | Multispectral demosaicking using adaptive kernel upsampling[END_REF], (b) Vis-NIR8[START_REF] Thomas | Spectral Characterization of a Prototype SFA Camera for Joint Visible and NIR Acquisition[END_REF], (c) IMEC16 and (d) IMEC25[START_REF] Geelen | A compact snapshot multispectral imager with a monolithically integrated per-pixel filter mosaic[END_REF]. Numbers are band indices. Vis5 band labels (Or=Orange, Cy=Cyan) are those of[START_REF] Monno | Multispectral demosaicking using adaptive kernel upsampling[END_REF] but could also be replaced by indices. Dominant G band is marked as white.

Fig. 3 .

 3 Fig. 3. (a) IMEC16 MSFA and (b) its S 2 pixel subset. Dashes on (a) bound the basic pattern of Fig. 1c.

Fig. 4 .

 4 Fig. 4. Neighborhood N p defined by the support N 8,1 for two pixels p ∈ S 2 (bold squares) in (a) IMEC16 and (b) VisNIR8 MSFAs, with associated bands B 2 shown in solid circles.

  . In the IMEC16 MSFA of Fig. 4a, the neighbors of any pixel p ∈ S 2 are associated with the bands B 2 = {12, 10, 9, 4, 1, 8, 6, 5} and |N 2,l p | = 1 for all l ∈ B 2 , where | • | is the cardinal operator. In the VisNIR8 MSFA of Fig. 4b, we have B 2 = {4, 7, 3, 5, 6, 8} and |N 2,l p | = 1 for l ∈ {5, 6, 7, 8}, but |N 2,3 p | = |N 2,4 p | = 2.

Fig. 5 .

 5 Fig. 5. Normalized SSFs of IMEC16 (left) and IMEC25 (right) cameras. Captions: band center wavelengths {λ k } K k=1 in ascending order.

  ) combines the spectral information of B MSFA(p) , i.e, the different bands that are associated with the neighbors of p. Because this set of bands depends on the band MSFA(p) associated with p, we separately consider each pixel subset S k to compute the LBP histogram. Specifically, we compute the histogram of MLBP[I raw ] for each band k ∈ {1, ..., K}: h k [MLBP [I raw ]]: [0, 2 P -1] → {0, . . . , |S k |} j → p ∈ S k , MLBP [I raw ] (p) = j . (15) Let us point out that only pixels in S k are considered to compute the k-th histogram. The concatenation of all the K histograms provides the final feature of size K • 2 P .

Fig. 6 .

 6 Fig. 6. Samples of HyTexiLa texture images rendered in sRGB under E (top), D65 (middle) and A (bottom row) illuminants, from left to right: vegetation, stone, food, textile, and wood.

Figure 7

 7 also shows that the number of bands available in the neighborhood of a pixel p generally depends on the distance d for a given MSFA. This number is formalized by |B k |, where k = MSFA(p) ∈ {1, . . . K} is the band associated with p (i.e., p ∈ S k ), and its dependency upon d is summarized in Tab. 1. In VisNIR8 for instance (see Fig.7b), the neighborhood of p ∈ S 3 contains eight different bands for d = 1 and d = 3, but only the bands 3 and 4 for d = 2. |B k | is also lower for d = 2 with Vis5 and IMEC16 MSFAs but is constant to eight whatever d ∈ {1,

Fig. 7 .

 7 Fig. 7. Neighborhood N p of a pixel p (bold square) in (a) Vis5, (b) VisNIR8, (c) IMEC16, and (d) IMEC25 MSFAs, considering the supports N 8,1 (solid circles), N 8,2 (dashed), and N 8,3 (dotted).

  (a) Vis5 (K = 5), (b) VisNIR8 (K = 8), (c) IMEC16 (K = 16), and (d) IMEC25 (K = 25).

Fig. 8 .

 8 Fig. 8. Classification accuracy (%) vs. computation cost (number of operations per pixel) of the different approaches (with D65 illuminant, d = 1, and WB demosaicing) for each MSFA: (a) Vis5 (K = 5), (b) VisNIR8 (K = 8), (c) IMEC16 (K = 16), and (d) IMEC25 (K = 25).

Table 1 .

 1 Number of available bands |B k |, k ∈ {1, . . . , K}, in the neighborhood of any pixel according to each MSFA and each distance.

	MSFA	d = 1 d = 2 d = 3
	Vis5	3 or 5 1 or 2 3 or 5
	VisNIR8	6	2	6
	IMEC16	8	3	8
	IMEC25	8	8	8

Table 2 .

 2 Feature size (histogram size and number of concatenated histograms) and required number of operations per pixel for each approach according to the number K of spectral bands. the shape of illuminant A that increases with respect to the wavelength, which makes pixel values in B and Cy channels significantly lower than in R and Or. Besides, values of pixels in S B ∪ S Cy are always compared to neighboring pixels in S Or ∪ S R (or vice versa) when d ∈ {1, 3}, whereas R values are compared to Or values and B values to Cy values when d = 2 (see Fig.

	Approach	Eq.	Feature size Hist. size Number of hist. Demosaicing Feature computation Number of operations
	Marginal LBPs [17]	Eq. (1)		K		24K
	Moment LBPs [19]	Eq. (3), Eq. (4)		3		K 2 + 10K + 69
	maLBP [20]	Eq. (5)		K + 1		17K + 8
	mdLBP [20] Cusano LBPs [21]	Eq. (6) Eq. (8)	2 P	2 K 2	4K -4	2 K + 24K + 7 15K + 27
	Lee LBPs [22] OBLBP [23]	Eq. (9) Eq. (10)		1 + K(K -1) K 2		27K 2 -26K + 24 24K 2
	IOBLBP [24]	Eq. (11)	2 P+1	K 2		27K 2 + 10K
	MLBP (ours)	Eq. (14), Eq. (15)	2 P	K	0	24
	lower accuracies by

Table 3 .

 3 Classification accuracy (%) of the different approaches for each experimental setting (illuminant, neighborhood distance, demosaicing method) and each MSFA:

  PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID

	Marginal LBPs [17] 87.71 86.95 88.39 87.50 88.67 88.46 87.36 87.23 88.60 88.26 88.46 88.60 84.89 84.62 87.71 87.84 88.05 88.19
	Moment LBPs [19] 90.11 88.05 88.26 88.67 86.74 86.13 90.32 88.53 89.29 89.01 88.53 87.36 87.98 87.16 86.74 86.74 85.99 85.71
	maLBP [20]	82.83 84.55 86.06 86.40 85.65 86.95 81.11 82.01 84.34 84.89 85.30 86.13 81.66 82.21 85.99 86.20 86.88 88.32
	Cusano LBPs [21] 87.64 87.36 88.60 88.26 88.80 89.15 86.88 87.36 88.46 88.39 89.29 89.22 88.80 89.01 90.52 90.25 90.32 90.87
	Lee LBPs [22]	93.41 93.75 95.40 95.12 95.60 94.85 92.99 93.13 94.57 94.44 95.47 95.47 91.48 91.96 94.23 93.82 94.64 94.37
	OBLBP [23]	97.39 97.46 97.39 97.53 97.12 97.46 96.84 96.70 97.18 97.39 97.12 97.39 94.64 94.51 95.67 95.95 96.29 96.57
	IOBLBP [24]	96.02 95.95 95.60 96.29 96.70 96.09 95.05 94.57 95.81 96.15 96.36 96.15 91.55 92.24 93.06 93.41 94.57 94.09
	MLBP (ours)	93.82	92.86	94.99	94.92	92.45	96.63	84.48	92.99	89.90
	(b)

96.91 96.63 96.57 96.57 96.63 96.09 96.22 96.15 95.67 96.09 95.54 95.19 95.40 95.47 95.12 95.12 95.33

  PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID Marginal LBPs [17] 84.89 86.40 85.44 85.78 86.06 86.13 87.09 88.94 87.50 88.67 87.64 88.53 84.00 85.51 85.03 85.37 85.78 85.51 Moment LBPs [19] 86.26 86.40 85.85 85.37 86.06 84.75 84.89 84.62 85.71 84.55 85.51 85.23 82.76 83.04 83.93 83.79 84.20 83.79 maLBP [20] 77.75 77.88 79.46 79.33 82.55 79.95 77.54 79.33 80.22 81.11 79.95 79.95 78.23 76.99 79.12 78.78 81.46 80.29 Cusano LBPs [21] 82.49 82.55 83.93 83.72 84.34 85.03 85.58 85.51 87.16 87.09 87.16 87.57 80.77 80.43 82.42 82.01 82.49 83.17 IOBLBP [24] 94.09 93.61 94.64 94.30 94.44 94.57 94.30 94.02 93.82 93.61 94.23 93.48 92.10 93.34 92.65 92.31 92.86 92.51 WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID Marginal LBPs [17] 81.59 85.10 83.79 86.54 83.86 86.20 81.11 84.89 83.72 86.95 85.30 87.16 80.70 84.89 83.10 85.92 85.71 87.50 Moment LBPs [19] 85.30 89.56 87.50 90.32 87.71 90.04 84.41 88.67 88.12 89.42 87.02 89.90 84.34 88.05 88.05 89.84 88.05 89.15 maLBP [20] 74.93 78.57 75.21 76.17 76.51 78.57 73.56 79.12 75.41 77.13 75.34 80.08 75.07 77.13 74.52 73.97 76.51 74.86 Cusano LBPs [21] 84.07 86.40 86.20 87.98 85.92 87.98 85.30 86.20 86.33 87.91 86.54 87.02 85.44 86.61 87.02 88.19 87.16 88.67 Lee LBPs [22] 89.77 95.40 92.45 96.02 93.68 95.95 89.22 95.26 92.58 95.95 93.41 95.81 87.09 94.44 90.93 94.09 92.93 94.64 OBLBP[START_REF] Mäenpää | Separating color and pattern information for color texture discrimination[END_REF]

	Lee LBPs [22]	91.28 91.69 93.96 93.89 95.05 95.26 89.35 91.41 91.83 91.69 93.48 93.41 91.62 92.86 93.82 93.89 95.26 94.16
	OBLBP [23] 96.29 MLBP (ours) 93.75	95.67	94.57	94.57	94.71	94.64	93.06	94.02	93.20
	(c)
			E			D65			A
	Approach	d = 1	d = 2	d = 3	d = 1	d = 2	d = 3	d = 1	d = 2	d = 3

96.91 96.57 96.22 96.77 96.09 96.50 97.25 97.60 97.05 97.05 96

  .84 96.98

95.67 96.09 95.47 95.81 95.05 95.95

	IOBLBP [24]	95.40 96

.36 95.19 95.88 95

  .33 95.47 96.02 96.91 95.88 96.57 95.74 96.29 93.54 94.71

94.44 94.78 93

  WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID WB PPID Marginal LBPs [17] 73.15 78.57 74.93 78.71 75.62 78.50 73.63 78.71 75.76 80.01 77.47 79.81 72.87 78.09 74.45 78.43 75.82 77.54 Moment LBPs [19] 79.46 83.17 80.56 83.72 80.77 84.27 78.98 84.20 81.25 85.51 81.46 85.16 78.71 83.31 80.15 83.93 80.36 84.07 maLBP [20] 63.67 78.64 61.26 78.50 67.03 76.79 68.27 78.09 62.77 77.06 65.73 75.14 64.08 77.88 60.37 78.37 67.17 75.55 Cusano LBPs [21] 77.13 81.80 78.57 82.69 77.68 81.39 75.34 80.15 76.51 82.35 78.98 82.14 76.65 81.18 78.64 81.94 77.75 80.70 .37 91.83 93.68 91.21 92.58 94.09 94.92 93.82 95.33 92.79 94.85 92.51 93.48 91.35 93.54 90.73 92.24

										.61 94.64
	MLBP (ours)	96.22	90.87	96.29	97.32	94.92	97.05	95.33	90.32	95.19
	(d)
			E			D65			A
	Approach	d = 1	d = 2	d = 3	d = 1	d = 2	d = 3	d = 1	d = 2	d = 3
	Lee LBPs [22]	87.77 92.58 92.03 93.89 92.79 94.57 89.49 94.16 91.14 94.78 92.31 94.92 88.12 91.96 91.96 93.54 92.65 94.23
	OBLBP [23]	94.02 95.05 93.41 94.92 93.48 94.71 94.99 95.88 94.51 95.95 94.51 95.60 93.75 94.51 93.13 94.71 93.06 94.16
	IOBLBP [24] 92.93 94MLBP (ours) 94.30	93.13	92.03	95.40	95.33	94.78	94.71	91.76	92.24

FUNDING

National French platform ANR-11-EQPX-23 IrDIVE; National French project CNRS-INS2I-JCJC-2017 MOSAIC.


Attribute-value rule learning (C ) (A 1 ) (A 2 ) (A 3 ) (A 4 ) (A 5 Task: induce rules to predict the value of the class attribute (C ) Rules extracted by Algorithm CN2

π CN2 1 : A 5 = Downtown ⇒ C = expensive π CN2 2 : A 2 < 2.50 ∧ A 4 = Toulouse ⇒ C = low-priced π CN2 3 : A 1 > 36.00 ∧ A 3 = D ⇒ C = low-priced
Interpretability of rules and rulesets

The logical structure of a rule can be easily interpreted by users

IF conditions THEN class-label

Rule learning algorithms generate rules according to implicit or explicit principles1 are the generated rules the interpretable ones? would it be possible to have dierent rulesets? why a ruleset would be better than another one from the interpretability point of view?

⇒ We need ways to analyze the interpretability of the outputs of rule learning algorithms

Analyzing the interpretability of rules Analyzing the interpretativeness of ruleset Impact of examples generalization on rule interpretability

(C ) (A 1 ) (A 2 ) (A 3 ) (A 4 ) (A 5 ) (A 6 )
: A 5 = Downtown ⇒ C = expensive π CN2 2 : A 2 < 2.50 ∧ A 4 = Toulouse ⇒ C = low-priced π CN2 3 : A 1 > 36.00 ∧ A 3 = D ⇒ C = low-priced
Impact of examples generalization on rule interpretability

(C ) (A 1 ) (A 2 ) (A 3 ) (A 4 ) (A 5 ) (A 6 )
: A 5 = Downtown ⇒ C = expensive π CN2 2 : A 2 < 2.50 ∧ A 4 = Toulouse ⇒ C = low-priced π CN2 3 : A 1 > 36.00 ∧ A 3 = D ⇒ C = low-priced
Toward the notion of admissibility The attribute-value model

A 1 A 2 A 3 A 4 A 5 A 6 A 7 • • • A n item 1 a 1,1 a 1,2 a 1,3 a 1,4 a 1,5 a 1,6 a 1,7 • • • a 1,n item 2 a 2,1 a 2,2 a 2,3 a 2,4 a 2,5 a 2,6 a 2,7 • • • a 2,n item 3 a 3,1 a 3,2 a 3,3 a 3,4 a 3,5 a 3,6 a 3,7 • • • a 3,n item 4 a 4,1 a 4,2 a 4,3 a 4,4 a 4,5 a 4,6 a 4,7 • • • a 4,n item 5
a 5,1 a 5,2 a 5,3 a 5,4 a 5,5 a 5,6 a 5,7 • • • a 5,n item 6 a 6,1 a 6,2 a 6,3 a 6,4 a 6,5 a 6,6 a 6,7

Subsets of data to generalize

A 1 A 2 A 3 A 4 A 5 A 6 A 7 A 8 A 9 item a 1,1 a 1,2 a 1,3 a 1,
4 a 1,5 a 1,6 a 1,7 a 1,8 a 1,9 item a 2,1 a 2,2 a 2,3 a 2,4 a 2,5 a 2,6 a 2,7 a 2,8 a 2,9 item a 3,1 a 3,2 a 3,3 a 3,4 a 3,5 a 3,6 a 3,7 a 3,8 a 3,9 item a 4,1 a 4,2 a 4,3 a 4,4 a 4,5 a 4,6 a 4,7 a 4,8 a 4,9 item a 5,1 a 5,2 a 5,3 a 5,4 a 5,5 a 5,6 a 5,7 a 5,8 a 5,9 item a 6,1 a 6,2 a 6,3 a 6,4 a 6,5 a 6,6 a 6,7 a 6,8 a 6,9 item a 7,1 a 7,2 a 7,3 a 7,4 a 7,5 a 7,6 a 7,7 a 7,8 a 7,9 item a 8,1 a 8,2 a 8,3 a 8,4 a 8,5 a 8,6 a 8,7 a 8,8 a 8,9 item a 9,1 a 9,2 a 9,3 a 9,4 a 9,5 a 9,6 a 9,7 a 9,8 a 9,9 ⇒ "Square" = selection of rows and columns in the data Rules A rule π expresses constraints (for a generic item x) which lead to conclusion C (x) (class which the item belongs to)

π : A 1 (x) ∈ v π 1 ∧ • • • ∧ A n (x) ∈ v π n → C (x) ∈ v π 0 ( * ) where v π i ⊆ Rng A i pour i = 1, . . . , n, v π 0 ⊆ Rng C . attributes i = 1, . . . ,
A 0 = 2 ∧ A 4 = Toulouse ⇒ C = low-priced f (C ) (A1) (A2) (A3) (A4) (A5) (A6)
S 0 = {low-priced} S 1 = {32, 40, 70, 75} S 2 = {2, 4} S 0 S 1 S 2
For every attribute A i , S i is the set of values of A i in items of S Each superset of S i is, theoretically speaking, a generalization of S i

Eliciting a rule (f function)

(A 0 ) (A 1 ) (A 2 )
Price Area Rooms 1 low-priced 70 2 2 low-priced 75 4 4 low-priced 32 2 7 low-priced 40 2

S 0 = {low-priced} S 1 = {32, 40, 70, 75} S 2 = {2, 4} S 0 S 1 S 2 f f
For every attribute A i , S i is the set of values of A i in items of S Each superset of S i is, theoretically speaking, a generalization of S i The generalisation process thus consists in selecting one of these supersets:

f choice function that is given as input a collection of supersets of S i and picks one

We are looking for an appropriate • for () i.e.

A 1 (x) ∈ S 1 ∧ • • • ∧ A n (x) ∈ S n → C (x) ∈ S 0 ()
Eliciting a rule (f function)

(A 0 ) (A 1 ) (A 2 )
Price Area Rooms 1 low-priced 70 2 2 low-priced 75 4 4 low-priced 32 2 7 low-priced 40 2

S 0 = {low-priced} S 1 = {32, 40, 70, 75} S 2 = {2, 4} S 0 S 1 S 2 f f
For every attribute A i , S i is the set of values of A i in items of S Each superset of S i is, theoretically speaking, a generalization of S i The generalisation process thus consists in selecting one of these supersets:

f choice function that is given as input a collection of supersets of S i and picks one

We are looking for an appropriate • for () i.e.

A 1 (x) ∈ S 1 ∧ • • • ∧ A n (x) ∈ S n → C (x) ∈ S 0 () Generalization of S i : S i = f ({Y | S i ⊆ Y ⊆ Rng A i })
Weakening closure operators

List of Kuratowski's axioms [Kur14] (closure system): Illustrations of the behaviour of CN2 Expected rules assuming capping or pre-closure for each class:

∅ = ∅ S ⊆ S ⊆ Rng A i S =
• topmost dataset: • v ∈ [-∞ : 15] ⇒ A 0 = blue • v ∈ [10 : +∞] ⇒ A 0 = green • bottom dataset: • v ∈ [-∞ : 15] ⇒ A 0 = blue • v ∈ [0 : +∞] ⇒ A 0 = green

  a rule A 1 = {75} ⇒ C = low-priced Most generalizing rule A 1 = [32 : 75] ⇒ C = low-priced Would the following rule be better? A 1 = [32 : 40] ∪ [70 : 75] ⇒ C = low-a rule A 1 = {75} ⇒ C = low-priced Most generalizing rule A 1 = [32 : 75] ⇒ C = low-priced Would the following rule be better? A 1 = [32 : 40] ∪ [70 : 75] ⇒ C = low-priced ⇒ this is the question of admissibilitya rule A 1 = {75} ⇒ C = low-priced Most generalizing rule A 1 = [32 : 75] ⇒ C = low-priced Would the following rule be better? A 1 = [32 : 40] ∪ [70 : 75] ⇒ C = low-priced ⇒ this is the question of admissibility! The notion of admissibility has to capture an intuitive notion of generalization . . .

  S S ∪ S = S ∪ S (pre-closure) Actually, we downgrade Kuratowski's axioms as follows S ⊆ S whenever S ⊆ S (closure) S = S whenever S ⊆ S ⊆ S (cumulation) S ∪ S ⊆ S whenever S ⊆ S (capping) Lemma: Kuratowksi ⇒ closure ⇒ cumulation ⇒ capping

Figure :

 : Figure: Distributions of the data for the classes blue and green.

Figure :

 : Figure: Distributions of the data for the classes blue and green.

  m a m,1 a m,2 a m,3 a m,4 a m,5 a m,6 a m,7 • • • a m,n Rows: items x 1 , x 2 , . . . , x m Columns: attributes A 1 , A 2 , . . . , A n ∀i, j a j,i ∈ Rng A iRng A i denotes the set of possible values for attribute A i

		• • • a 6,n
	item 7 . . .	a 7,1 a 7,2 a 7,3 a 7,4 a 7,5 a 7,6 a 7,7 • • • a 7,n . . . . . . . . . . . . . . . . . . . . . . . . . . .

item

  The logical structure of rules makes them easy to read but ... The interpretability of rules learned from examples requires, in particular, to take care of the way examples are generalized Example of numerical attributes, but also symbolic attributes with structures (e.g. orders) Qualifying the interpretable nature of rule learning outputs is challenging What can our approach do for rule interpretability Our work contributes by giving a way to do such analysis A proposal of a general framework for rule learning A topological study of admissible generalisations of examples

principles mainly based on statistical properties!

Notion of admissibility: propositions

Generalization of S i :

What choice function(s) can in practice capture these expected algebraic properties? Notion of admissibility: propositions Generalization of S i :

X is a closure system upon Rng A i .

• is an operation enjoying weaker properties than closure operators; alternatives looked at:

pre-closure operator capping operator

What choice function(s) can in practice capture these expected algebraic properties?

Notion of admissibility: propositions Generalization of S i :

X is a closure system upon Rng A i .

• is an operation enjoying weaker properties than closure operators; alternatives looked at:

moreover, X is a collection containing all super-intervals of an interval belonging to the collection Class of choice functions satisfying pre-closure Theorem. Given a set Z , let f : 2 2 Z → 2 Z be a function st for every upward closed X ⊆ 2 Z and every

is a pre-closure operator upon Z .

Numerical attributes: principle of single point (u) interpolation

Class of choice functions satisfying capping

Theorem. Given a set Z , let f : 2 2 Z → 2 Z be a function st for every X ⊆ 2 Z such that X ∈ X and for every

is a capping operator upon Z .

Intuition: Z is Rng A i X (and Y, too) is a collection of intervals over Rng A i moreover, X is a collection whose intersection is itself a member of the collection Class of choice functions satisfying capping Theorem. Given a set Z , let f : 2 2 Z → 2 Z be a function st for every X ⊆ 2 Z such that X ∈ X and for every

is a capping operator upon Z . Rules learned by CN2, topmost dataset:

Numerical attributes: principle of pairwise point interpolation

Separating interesting intervals

Rules learned by CN2, bottom dataset: