Khadidja Ould Amer

Baptiste Magnier
email: baptiste.magnier@mines-ales.fr

Stefan Janaqi
email: stefan.janaqi@mines-ales.fr

Manuela Cesaretti
email: manuela.cesaretti@stellasurgical.com

Clement Labiche
email: clement.labiche@stellasurgical.com

Significant Smartphone Images Features For Liver Steatosis Assesment

Keywords: Liver images, histograms, texture, LASSO

The growing demand for liver transplantation is leading medicine to explore new technologies in order to extend the field of viable transplants. Automatic hepatic steatosis assessment is the first step towards the development of a computeraided liver diagnosis due to its importance as a risk factor for primary dysfunction. Color and texture are considered as two fundamental visual characteristics to assess the degree of hepatic steatosis. The aim of this study is to determine the discriminating features for liver image classification according to three steatosis classes: mild, moderate or severe. First, color features are extracted in three color spaces namely: RGB, HSV and YCbCr. Texture features are then extracted from RGB components using co-occurrence matrices, Local Binary Pattern (LBP) and Local Phase Quantization (LPQ). Additionally, feature-extraction was followed by generating a linear model regression using Least Absolute Shrinkage and Selection Operator (LASSO) allowing the classification task. The experimental results show that RGB histograms provide better classification accuracy. However, the relevance of these preliminary results is not sufficient to draw a decisive conclusion due to the experimental database, which is really small, very unbalanced and the images were captured under unknown and widely varying conditions.

I. INTRODUCTION AND CONTEXT

Liver transplantation (LT) provides indisputable benefit for the survival and quality of life of patients with end-stage diseases [START_REF] Martins | Hepatic steatosis assessment[END_REF]. The advances in liver transplantation medicine have led to its expansion over the past ten years. However, access to allografts remains the main limitation, because the growing number of patients who may benefit from the transplant consistently exceeds the number of organ donors. This issue has led to expand the liver selection criteria, which could have Hepatic Steatosis (HS), moving so the limit of acceptance for marginal livers. HS occurs initially by accumulation of triacylglycerols (TGs) in the cytoplasm of hepatocytes. It corresponds to the most important factors affecting liver allograft function and so LT outcome. Based on the size of hepatocytes TGs droplets, HS can be categorised as microvesicular or macrovesicular. Microvesicular has less influence on poor graft function than macrovesicular (MaS) which is one of the most important criteria defining the extended criteria of donor organs. Current quantification and grading of HS utilize a classification of mild (<33%), moderate (33%-66%), or severe (>66%) [START_REF] Cesaretti | Assessment of liver graft steatosis: Where do we stand?[END_REF]. Mild MaS is considered a transplantable condition, whereas the use of donor livers with moderate MaS remains challenging. Grafts with severe steatosis (>60%) are highly recommended to be excluded, due to the critical risk of primary non-function.

Currently, the histological analysis of biopsied liver tissue performed during or after procurement is considered as the gold reference standard for the HS evaluation. However, this requires the availability of a cryostat facility and of an expert pathologist to interpret the result before or immediately after the liver retrieval. Consequently, this procedure can be invasive, time-consuming and difficult in a remote donor hospital. Due to the short time ailability between the liver procurement and its transplantation, the surgeons usually perform HS assessment through clinical evaluation (medical history, blood tests) and qualitative visual liver assessment [START_REF] Cesaretti | Assessment of liver graft steatosis: Where do we stand?[END_REF]. The liver is evaluated in a standardized fashion according to certain criteria of parenchymal texture such as: yellowness, firmness and round liver edges. Fig. 1 shows two livers belonging to two different hepatic steatosis (HS) classes assessed by histopathological examination of liver tissue samples extracted with biopsy. Thus, liver with high percentage of steatosis (Fig. 1(right)) is characterized by yellowish colors and more rounded edges than the liver in Fig. 1(left) with mild hepatic steatosis (HS=5%). Such indicators can help surgeons to assess the overall liver quality for transplantation. Even if biopsy samples are taken, visual assessment of liver steatosis remains very important and always performed. However, it depends on the experience of the surgeon and becomes sometimes very difficult even for experienced surgeons [START_REF] Martins | Hepatic steatosis assessment[END_REF]. In this context, the development of a simple, handy, fast and accurate device for non-invasive quantification of HS would obviously be most desirable.

As smartphones have become ubiquitous and the new generation ones are equipped by high resolution cameras, they could be the solution for automatic HS assessment. Indeed, the recent advances in computer vision has led to the development of various assisted diagnostics systems. In this context, texture and color are the fundamental characteristics used for optical images classification [START_REF] Sumithra | Segmentation and classification of skin lesions for disease diagnosis[END_REF]- [START_REF] Pang | Quantitative evaluation methods of skin condition based on texture feature parameters[END_REF]. Inspired by these promising studies, the aim of this paper is to study different image descriptors in order to find the relevant characteristics allowing to predict the quality of the liver for transplantation and its limitations. This project represents the first attempt Fig. 1: Images of two livers acquired with smartphones in the operating room: liver with mild: HS=5% (left) and with severe hepatic steatosis: HS=85% (right).

to use automatic texture analysis of RGB (for Red Green and Blue channels) images and machine learning for graft HS assessment. The image database used in this study was initially introduced in the reaserch of Moccia et al. [START_REF] Moccia | Computerassisted liver graft steatosis assessment via learning-based texture analysis[END_REF] on the same project. These images were acquired with different smartphones under different conditions wich include: a wide range of illumination, varying camera pose, different organ positions and presence of specular reflections. There is also the problem of image demosaicking [START_REF] Losson | Comparison of color demosaicing methods[END_REF] or image compression, degrading textures, colors and many details.

Features extraction consists to apply descriptors on the liver image. It is therefore interesting to know the precise region of the lobe in which the experts perform the biopsy. In this case, it makes more sense to apply the descriptors to the patch covering the region to be analyzed. However, the complete liver shape in many images of the database are not available. Consequently, the classification feasibility will be evaluated by applying both texture and color descriptors on the whole visible area of the liver in this study.

II. TEXTURE DESCRIPTORS

Texture analysis is a useful way whose primary purpose is to provide descriptors for image classification. In the medical field, texture-based information are increasingly used for computer aided diagnosis [START_REF] Moccia | Computerassisted liver graft steatosis assessment via learning-based texture analysis[END_REF], [START_REF] Riaz | Detecting melanoma in dermoscopy images using scale adaptive local binary patterns[END_REF], [START_REF] Kavitha | Texture and color feature extraction for classification of melanoma using svm[END_REF]. In this study, we focused on three commonly used texture descriptors: Cooccurrence matrices, Local Binary Pattern (LBP) and Local Phase Quantization (LPQ).

A. Gray Level Co-occurence Matrix (GLCM)

Grey Level Co-occurrence Matrix is one of the earliest methods used for image texture analysis [START_REF] Haralick | Textural features for image classification[END_REF]. This technique calculates the probability P (i, j, d, θ) which represents the number of times two pixels with gray level i and j appear at a relative distance d according to a given orientation θ:

P i,j (d, θ) = # [(k, l)(m, n) ∈ (Lx, Ly) 2 | I(k, l) = i, I(n, m) = j, d, θ] , (1)
where (k, l) and (m, n) are respectively, the coordinates of pixels i and j, # denotes the occurrence number of the pixel pair in the (L x ×L y ) image resolution cells ordered by their row-column designations. The conventionally used angular directions θ are 0, 45, 90 and 135 degrees. Note that P (i, j, d, θ)=P (j, i, d, θ), thus GLCMs are a symmetric.

GLCMs provide a rich description of spatial dependence that is difficult to manipulate directly. Haralick et al. proposed a set of 14 statistical descriptors or attributes to summarize the textural information contained in the GLCMs [START_REF] Haralick | Textural features for image classification[END_REF]. However, only a subset of 5 descriptors are considered to be more relevant in the literature, namely: angular second moment or energy EN E, contrast CST , homogeneity HOM , entropy EN T and correlation COR are computed such as:

EN E = i j P i,j (d, θ) 2 , ( 2
)
CST = i j (i-j) 2 • P i,j (d, θ), ( 3
)
HOM =- i j P i,j (d, θ) 1+(i-j) 2 , ( 4
)
EN T = - i j P i,j (d, θ) • log(P ij (d, θ)), ( 5
) COR = - i j P i,j (d, θ) (i -μ i )(j -μ j ) σ 2 i σ 2 j , ( 6
)
where μ i and σ j represent the mean and standard deviation of the normalized inputs for the reference pixel i and neighbor pixel j respectively [START_REF] Haralick | Textural features for image classification[END_REF]. Also, μ {i,j} and σ {i,j} correspond to

μ i = i i • P i,j and σ i = i P i,j (i -μ i ) 2 .

B. Local Binary Pattern (LBP)

The original version of the LBP labels the pixels by comparing each pixel with its 3×3-neighbourhood and considering the result as a binary number as illustrated in Fig. 2 [START_REF] Ojala | A comparative study of texture measures with classification based on featured distributions[END_REF]. The histogram of the labels is often used as a texture descriptor. The LBP operator was later generalized to deal with the texture at different scales by using neighborhoods of different sizes [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF]. The thresholding function is defined as follows:

LBPP,R = P -1 p=0 s(gp -gc) • 2 p , with s(x) = 1 x ≥ 0 0 x < 0, (7)
where g c and g p are the gray level values of the central pixel and its neighbor respectively. P is the total number of the sampling points in a circular neighborhood of R radius around g c . The LBP P,R operator can produce 2 P different output values from the P neighbor pixels. Furthermore, the LBP operator rotation invariant LBP ri P,R by assigning a unique label to each rotation invariant [START_REF] Pietikäinen | Rotation-invariant texture classification using feature distributions[END_REF] (as an example, 01110000 and 00111000 are rotation invariant) is expressed by:

LBP ri P,R = min {ROR(LBPP,R, i) | i = 0, 1, ..., P -1}, (8)
where R OR (x, i) performs a circular bit-wise right shift on the P-bit number x, i times.

Another extension of the LBP operator has been proposed to improve both the rotational invariance and to reduce the feature dimensionality. The uniform rotational invariance LBP riu2 P,R assigns an individual code to uniform patterns which contain at most 2 transitions (0 to 1 or vice versa) and a separate code to all non-uniform patterns. Its formal definition is given by the following equation: where U (LBP P,R ) is computed as:

LBP riu2 P,R = P -1 p=0 s(gp -gc) if U (LBPP,R) ≤ 2 P + 1 otherwise, (9)
U (LBP P,R ) = |(s(g P -1 -gc) -s(g 0 -gc)) | + P -1 p=1 |(s(gp -gc) -s(g p-1 -gc)) |.
Thereafter, the LBP riu2 P,R operator is a texture descriptor by producing a histogram of (P +2) bins.

C. Local Phase Quantization (LPQ)

The LPQ is blur invariant texture analysis method based on binary encoding of the phase information extracted from the 2-D short-term Fourier transform (STFT) computed over a local neighborhood at each pixel position of the image [START_REF] Ojansivu | Blur insensitive texture classification using local phase quantization[END_REF]. Thus, in digital image processing, the spatial blurring is represented by a convolution between the image intensity and a point spread function (PSF). In the Fourier domain, for given coefficient computed by the STFT u, this corresponds to: G(u) = F (u) • H(u), with G, F , and H represent the Fourier transforms of the blurred image, the original image, and the PSF respectively. By considering only the phase of the spectrum, the relation turns into a sum:

∠G(u) = ∠F (u) + ∠H(u). ( 10
)
If the blur PSF is centrally symmetric, the Fourier transform H(u) will always be real valued, and, as consequence: ∠H(u) = 0 if H(u) ≥ 0, π otherwise. Furthermore, the shape of H(u) for a regular PSF is close to a Gaussian or a sincfunction, which makes at least the low frequency values of ∠H(u) to be positive [START_REF] Heikkilä | Local phase quantization for blur insensitive texture description[END_REF]. Therefore, at these frequencies, ∠G(u) = ∠F (u) causing ∠F (u) to be a blur invariant. The LPQ method is based on this property of blur invariance. The local phase information is extracted using STFT computed over a rectangular neighborhood N x of size M ×M at each pixel position x of the image f (x):

F (u, x) = y∈Nx f (x -y)e -j2πu T y . ( 11
)
Only four (more informative) complex coefficients are considered in LPQ, corresponding to: T , where a is small enough to satisfy H(u)≥0. This results in a 1×4 vector for each pixel position:

u 1 =[a, 0] T , u 2 =[0, a] T , u 3 =[a, a] T , u 4 =[a, -a]
F x = [F (u 1 , x), F (u 2 , x), F (u 3 , x), F (u 4 , x)]. ( 12
)
Finally, the phase information in the Fourier coefficients is recorded by observing the signs of the real and imaginary parts of each component in F x as follows:

q j (x) = 1 if g j (x) ≥ 0 0 otherwise, ( 13
)
where g j (x) is the j th component of the vector

G x = [Re{F x }, Im{F x }].
The resulting eight binary coefficients q j (x) are represented as integer values between 0-255 using a binary encoding:

f LP Q (x) = 8 j=1 q j (x) • 2 j-1 . ( 14
)
Finally, as for the LBP descriptor (Fig. 2), the histogram of the label image f LP Q is used as a 256-dimensional feature vector in classification.

An improved rotation-invariant local phase quantization (RI LP Q) approach was proposed in [START_REF] Ojansivu | Rotation invariant local phase quantization for blur insensitive texture analysis[END_REF]. This method consists of two steps: characteristic orientation estimation and directed descriptor extraction. Feature computation procedure is similar to the original LP Q, but the neighbourhood at each pixel location is rotated according to the direction of the characteristic orientation. Consequently, this technique (RI LP Q) produces 256 features for a given grey level image and is compared in this study with previous descriptors.

III. COLOR DESCRIPTORS

Since hepatic steatosis causes a change in the color of the liver, color information becomes interesting features for image classification. Indeed, it has been demonstrated that considering the color in the image significantly improves the performance of many tasks in medical image analysis such as: melanoma detection [START_REF] Riaz | Detecting melanoma in dermoscopy images using scale adaptive local binary patterns[END_REF] and skin lesions classification [START_REF] Marques | On the role of texture and color in the classification of dermoscopy images[END_REF], [START_REF] Rahman | Skin lesions classification based on color plane-histogramimage quality analysis features extracted from digital images[END_REF]. Several color representations have been proposed to interpret the color information, each might outperform others in a particular application. This section introduces the commonly used color spaces in medical image analysis and presents the main texture descriptors trough these color spaces which are useful for the proposed study.

A. Color Histogram Features

RGB is the default color space for storing and representing digital images. In this space, each color appears in its primary color components: Red, Green and Blue. There are many other color spaces that can be computed from the RGB space by means of linear or nonlinear transformations [START_REF] Truong | Multi color space lbp-based feature selection for texture classification[END_REF]. The HSV color space is closer to how humans perceive color in terms of: Hue (H), Saturation (S) and value (V). The hue component describes the wavelength of the color, saturation indicates the amount of white in the color and value represents the brightness which describes the intensity of light coming from the color. Several researches have used the HSV space to interpret the color information in color medical images. Indeed, the hue can reflect the abnormal appearance of tissues, saturation can be used to assess the level of abnormalities and value may indicates the property of luminance [START_REF] Feng | A machine learning texture model for classifying lung cancer subtypes using preliminary bronchoscopic findings[END_REF]. YCbCr is another color model which represents color. The Y in YCbCr signifies the luminance component while C b and C r represent the blue-difference and red-difference chroma components. YCbCr color model imitates human eye which is more sensitive to light intensity changes than hue changes. Therefore, the model allows to perceive more information from luminance and to obtain an efficient image representation [START_REF] Soleimanizadeh | Recognition of partially occluded objects based on the three different color spaces (rgb, ycbcr, hsv)[END_REF].

Regardless of the used model, the distribution of intensities in each color channel is represented by a histogram of 256 bins (for images coded on 8 bits). These histograms can be concatenated and used as color descriptor of (3 × 256) values [START_REF] Marques | On the role of texture and color in the classification of dermoscopy images[END_REF], [START_REF] Nahid | Histopathological breast-image classification using concatenated rgb histogram information[END_REF]. Instead of concatenating the three histograms independently, it is also possible to generate a 3D histogram representing the joint distribution of the three channels. However, this increases the space dimensionality to 256 3 and an image quantization becomes necessary [START_REF] Riaz | Detecting melanoma in dermoscopy images using scale adaptive local binary patterns[END_REF], [START_REF] Rahman | Skin lesions classification based on color plane-histogramimage quality analysis features extracted from digital images[END_REF].

B. Color and texture combination

Color and texture are two related characteristics of the image, but these characteristics are usually analyzed separately. Indeed, texture analysis methods are often applied to grayscale images for computational reasons and also for the interest of the intensity values of the pixels in the image. However, it has been demonstrated that texture features incorporating color information can improve the classification results. This generated two categories of methods:

• methods extracting color and texture features separately, • methods considering color and texture jointly [START_REF] Mäenpää | Classification with color and texture: jointly or separately?[END_REF]. In the first approach, texture features extracted from the grayscale image are used in conjunction with other features describing the color distribution in a given color space [START_REF] Riaz | Detecting melanoma in dermoscopy images using scale adaptive local binary patterns[END_REF], [START_REF] Kavitha | Texture and color feature extraction for classification of melanoma using svm[END_REF], [START_REF] Indriani | Tomatoes classification using k-nn based on glcm and hsv color space[END_REF]. For example, Riaz et al. [START_REF] Riaz | Detecting melanoma in dermoscopy images using scale adaptive local binary patterns[END_REF] combine texture features extracted from GLCM with color histograms computed in three color space (including RGB and HSV) to classify the dermoscopy images into melanoma and non-melanoma. In the second approach, spatial and color distribution of pixels are jointly characterized by evaluating the texture features within each of the three color components independently [START_REF] Ali | Symptom based automated detection of citrus diseases using color histogram and textural descriptors[END_REF], or by considering the spatial interactions within and between those components in a given color space [START_REF] Truong | Multi color space lbp-based feature selection for texture classification[END_REF].

In the following, the image classification is done in a first time by comparing histograms of concatenated color space channels (RGB, HSV, YCbCr), then by applying texture descriptors on the image in the most effective color space.

IV. EXPERIMENTS AND RESULTS

A. Experimental Database

The database used in this study contains 154 RGB images, which refer to 154 different liver acquired with open-surgery view. As mentioned in the introduction, images were acquired with smartphones by surgeons under various conditions, which makes the classification task very challenging. Dataset images are divided into three classes according to HS rate assessed with biopsy. There is a significant imbalance between the classes: 125 images in class 1 (HS ≤ 30%), referring to transplanted livers, 14 images in class 2 (30 < HS ≤ 50%) corresponding to livers with moderate steatosis and 17 images in class 3 (HS > 50%) referred to discarded grafts with severe steatosis. Note that, these classes can be optionally changed depending on the expert decision.

Image analysis consists of applying the descriptors on the global area of the liver. Thus, manual segmentation was performed on each image to separate the liver organ from the background as shown in Fig. 3(a). Shadow areas and specular reflections were removed by setting to zero the bottom 'lowPercentile' and the top 'highPercentile' of all pixel values of the image, see Fig. 3(c

B. Feature extraction

Image classification requires comparison of images according to a certain useful features. In this study, color and texture features were considered in order to find suitable representation for images. First, color features were extracted using color histograms in three color spaces namely: RGB, HSV, YCbCr. Indeed, the aim is to determine if a particular color space is optimum for our study.

Color content of the liver can be represented by a single 3D histogram or three separate 1D histograms. In a simplified way, for each color space, a feature vector is constructed by concatenating the corresponding three component histograms. We could have used the joint 3D histogram of the three channels since they are not statistically independent. However, this involves an image quantization to reduce the space dimensionality and thus a loss of information.

The differential structures in liver images may be analyzed with texture descriptors. In this study, spatial and color distribution of pixels were jointly characterized by evaluating texture features within each of the three color components (Red Green and Blue) of the image independently. Co-occurrence matrices were computed on these 3 channels separately using all the possible combinations of (θ, d), with θ∈{0 • , 45 • , 90 • , 135 • } and d∈{1, 2, 3} (see eq. 1); then, the corresponding Haralick five features (EN E, CST , HOM , EN T and COR) were concatenated. On the other hand, LBP P,R riu2 (eq. 9) were then computed on each of the three channels: 10 features from LBP 1,8 riu2 , 18 features from LBP 2,16 riu2 and 26 features from LBP 3,24 riu2 , wich gives a total of 3 × (10 + 18 + 26) LBP based features to a given image. Finally, rotation invariant RI LP Q were computed over three neighborhood sizes (M = 3;5;7) and the corresponding histograms were concatenated. Note that the values of M were chosen to be consistent with the scales d and P used to calculate co-occurrence and LBP features, respectively. A total of 3×(256×256) features were obtained by this method.

C. Classification with LASSO

Given a set of couples of observations (x k , y k ) k={1,....m} , with x k ∈R d , a classical linear regression algorithm looks for a w∈R d such that the predicting function f (x, b, w) = b + w T x fits the couple (x k , y k ). Let X∈R (m×d+1) be the matrix with rows [1, x T k ] k={1,....m} and w = [b, w]. When the quality of fitting is measured by Euclidean distance, the optimal w is the solution of the optimization problem:

min L(w) = 1 m Xw -y 2 2 , ( 15
)
where L is the loss function to be minimized. The d explaining features are the columns X c of X with c = 2, , d + 1. When a budget of C features is given, the optimal w is the solution of:

min L(w) = 1 m Xw -y 2 2 , ||w|| 0 C. ( 16
)
Here, ||w|| 0 = |{c | w c = 0, c ≥ 2}| (notice that ||w|| 0 is not a norm). The problem (eq. 16) arises also when the number of observations is less than the number of features m<d. Our problem falls into this case with m∼10 2 limited by the number of patients and d∼10 3 the number of numeric characteristics calculated from images. A main fact from linear algebra shows that the number of predicting features cannot be greater than rank(X) m. Thus, one searches for a subset of C m features that give the best fit for y. The problem (eq. 16) is NP-hard and a good heuristic is to solve the following:

min L(w) = 1 m Xw -y 2 2 , ||w|| 1 K. ( 17
)
The minimization problem (eq.16) is LASSO (Least Absolute Shrinkage and Selection Operator) [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]:

min L(w) = 1 2m X w -y 2 2 +λ • ||w|| 1 . ( 18
)
This is a convex optimization problem that often finds sparse solutions w. In fact, the * 1 -norm penalizing with the factor λ brings w c to 0 whenever corr(y, X c )<λ. The solution w gets sparser with increasing λ. Then, a trade-off can be easily found between the prediction error and the number of predicting features by controlling the sparsity of w through the penalizing factor λ.

To perform the classification, image database was divided into: a learning set (117 images) and a test set (37 images). Note that the learning set was created by randomly selecting 75% of images in each of the three classes while the remaining 25% were left to the test. For each of the features datasets, a learning model is constructed using LASSO classifier with the selected features. To evaluate the classification performance of the resulting models, sensitivity (Se), specificity (Sp) and accuracy (Acc) were compputed from the three confusion matrices (C1 vs C2C3), (C2 vs C1C3), (C3 vs C1C2):

Se = T P T P +F N , Sp = T N T N +F P , (19)
and

Acc = T P +T N T N +F P +T P +F N , ( 20
)
where C i,i∈{1,2,3} are the three steatosis classes. Each row of the confusion matrices represents the instances in a predicted class while each column represents the instances in an actual (Act.) class. T P , T N, F P and F N represent the number of true positive, true negative, false positive and false negative, respectively. The Tab. I reports all the classifications. In that respect, the best classification performance was obtained with the RGB dataset: (Se = 0.47, Sp = 0.74, Acc = 0.81). In this case, LASSO retains only 129 out of 3×256 features to drive features are really important to drive the correct predictions. However, by analysing the RGB confusion matrix, the sensitivity is decreased to (Se = 0.47) by the fact that the model correctly classified only one image of class 2 and 3. This problem is mainly due to the fact that we don't have enough images of classes 2 and 3 compared to class 1 in our database. This creates a significant imbalance between classes, reducing the model's ability to learn. For texture features, the results show that LBP perform better than co-occurrence and LPQ features with (Se = 0.42, Sp = 0.69, Acc = 0.78). However, there is no improvement compared to RGB results.

V. CONCLUSION

The aim of this study is to design a reliable model for automatic hepatic steatosis assessment. More specifically, we tried to answer the following question: which characteristics, texture or color, convey the most relevant information? The results showed that RGB histograms provide better classification accuracy. This contribution strengthens our claims that seeking specific descriptors might be the key for a future computer assisted system that can support surgeons to make the decision to accept or reject grafts. Future work should extend these results by using a larger and more balanced database for robust conclusions. Moreover, image acquisition needs to consider:

• camera pose • organ position, • color calibration of the imaging device, • incoming illumination parameters influencing the color measurements.

Also, it would be interesting to combine the stronger color and texture feature set to increase the performance of the system.

Fig. 2 :

 2 Fig. 2: Basic LBP and LPQ histogram calculation example.

Fig. 3 :

 3 Fig. 3: Liver image acquired by a surgeon smartphone.

  ).

			RGB		Cooccurrrence		HSV			YCbCr			LBP			LPQ
		Act.1 Act.2	Act.3 Act.1 Act.2	Act.3 Act.1 Act.2	Act.3 Act.1 Act.2	Act.3 Act.1 Act.2	Act.3 Act.1 Act.2	Act.3
	Predict. 1	25	1	3	20	3	2	22	3	0	18	3	2	23	3	2	22	3	3
	Predict. 2	0	1	0	5	0	2	6	0	4	7	0	1	3	0	0	7	0	1
	Predict. 3	5	1	1	5	0	0	2	0	0	5	0	1	4	0	2	1	0	0
			Metric			Metric			Metric			Metric			Metric			Metric
		Se	Sp	Acc	Se	Sp	Acc	Se	Sp	Acc	Se	Sp	Acc	Se	Sp	Acc	Se	Sp	Acc
	C1 vs C1C2	0.83	0.42	0.75	0.66	0.28	0.59	0.73	0.57	0.7	0.6	0.28	0.54	0.76	0.28	0.67	0.73	0.14	0.62
	C2 vs C1C3	0.33	1	0.94	0	0.79	0.72	0	0.7	0.64	0	0.76	0.25	0	0.91	0.83	0	0.76	0.7
	C3 vs C1C3	0.25	0.81	0.75	0	0.84	0.75	0	0.93	0.83	0.25	0.93	0.78	0	0.87	0.83	0	0.96	0.86
	Mean	0.47	0.74	0.81	0.22	0.64	0.69	0.24	0.73	0.72	0.28	0.63	0.67	0.42	0.69	0.78	0.24	0.62	0.72

TABLE I :

 I Performance measures for different color spaces and texture features.


Cédric Walbrecq
email: cedric.walbrecq@mines-ales.fr

Euromov Dhm

Isabelle Marc
email: isabelle.marc@mines-ales.fr

Baptiste Magnier
email: baptiste.magnier@mines-ales.fr

Dominique Lafon-Pham
email: dominique.lafon@mines-ales.fr

Video logarithmic tone mapping operator with automatic calculation of the logarithmic function base

Keywords: HDR, Video Tone Mapping, Temporal artifact reduction I

High Dynamic Range (HDR) videos are intended to capture the full range of luminance that can be found in a real world environment. But, as display devices with restricted luminance range are still the most common ones, the need for efficient tone mapping tools remains. Applying a variable base logarithmic compression curve on HDR values is a method that gives good results on static images. Video mapping requires further processing, in order to maintain correct contrast over the different frames and to minimize undesirable artifacts such as flicker and temporal incoherencies when the overall lighting conditions vary. This paper describes a new method for a real time automatic calculation of the logarithmic function base, depending on the characteristics of successive frames in a sequence. Compared to some well-known operators, our proposal better performs in terms of temporal coherency and overall contrast.

INTRODUCTION

Since the late 1990s, there is a growing interest in developing High Dynamic Range (HDR) imaging methods in order to capture, encode and render images or videos aiming to mimic real world scenes. However, most of the currently commercially available display devices offer a limited dynamic range of two orders of magnitude, well below the range of luminance between the lightest and darkest areas in natural viewing, so works still have to be done to develop and enhance tone mapping operators (TMO), i.e. methods to reduce luminance range while preserving details in dark and light areas as well as overall brightness perception. It is particularly true concerning TMO for videos, as the need for them has appeared later, with the more recent development of HDR video contents. HDR video processing requires a new set of tools, because the simple application of any image operator on each of the successive frames might led to perceptually unacceptable results, in case of failure in temporal correlation between two frames. We propose a method to extend an image operator for video processing, which adapts the tone mapping function to the temporal variations in lighting conditions and scene composition, and prevents the appearance of frames with incoherent luminance content. A comparison is made with others TMOs on a set of sequences with different luminance ranges and variations, and our proposal gives promising results.

II. RELATED WORKS

Tone mapping operators are essentially nonlinear functions: because of the nonlinear behavior of the Human Visual System (HVS), a simple linear scaling of HDR values into a reduced range does not allow for the matching between perception of the tone mapped image and perception of the real-world scene. Hundreds of image TMOs have been proposed in the last decades to meet different objective: fidelity to the HDR scene, with detail preservation in both dim and bright areas, simulation of human visual processing, for instance taking into account light and dark adaptation processes, or artistic rendering [START_REF] Eilertsen | A comparative review of tone-mapping algorithms for high dynamic range video[END_REF]. In terms of processing, they fall in two wide categories: global operators apply the same mapping to all pixels in the image, while local ones adapt the mapping function to each pixel depending on its neighborhood. Global and local operators both have advantages and limitations, and at this time, there is no clear evidence of one category being superior to the other. Rather, it appears that their relative performances depend on the way they are evaluated. Numerous psychophysical comparison studies (for a review, see for instance [START_REF] Cerda-Company | Which tonemapping operator is the best? A comparative study of perceptual quality[END_REF], have demonstrated that images processed by global methods are generally better ranked in terms of overall brightness, global contrast, and naturalness. On the contrary, local methods overperform in terms of detail preservation and local contrast.

TMOs designed for static images might obviously be directly applied on each frame of video sequences, but doing that might lead to unacceptable results, with temporal artifacts generation: image TMOs are designed to scale HDR luminance values on the whole range of a LDR display, and are often based on image statistics (i.e. mean, max, histogram, etc). Whenever the range of luminance or image statistics vary between two successive frames, because of a sudden difference in lighting conditions, or motion of the camera during walkthrough, it may break the temporal coherency between these frames (for instance: the second LDR frame appearing brighter though the second HDR one is darker). A detailed definition of these artifacts may be found in [START_REF] Boitard | Zonal brightness coherency for video tone mapping[END_REF] and will be detailed in the following. Several subjective quality assessment works have been performed on tone mapped videos. Melo et al focused their study on small mobile displays (tablet) and asked sixty participants to compare the subjective performances of six operators for images and videos tone mapping, in reference to an HDR display [START_REF] Melo | Evaluation of HDR video tone mapping for mobile devices[END_REF]. In [START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR Video[END_REF], five expert observers rated videos in term of perceptual attributes (overall brightness, overall contrast, overall color saturation, temporal color consistency, temporal flickering, ghosting and noise), for eleven TMOs. , a set of quantitative measurements was used to rank these TMOs: noise visibility, percentage of under and over exposed pixels, loss of contrast, and temporal coherence, which was derived from cross correlation between HDR and LDR values over a few successive frames. It can be drawn from these studies that subjective quality relies on different characteristics for images and videos: for the later, the preservation of contrast in fine details becomes much less important than the general impression of temporal coherence, and global operators are better ranked in general. Brightness flickering and ghosting effect are found to be critical factors.

For the TMOS that are intended to simulate the SVH processing, temporal effects are explicitly taking into consideration by modelling dark and light adaptation mechanisms: Fewerda et al scaled HDR values so as just noticeable contrast differences (JND's) in the real scene correspond to contrast JNDs in the mapped image, for the current adaptation state on each frame [START_REF] Ferwerda | A model of visual adaptation for realistic image synthesis[END_REF]. Pattanaik et al proposed a time dependent TMO, with adaptation process and image appearance modelling [START_REF] Pattanaik | Timedependent visual adaptation for fast realistic image display[END_REF]. On the other part, TMOs with no explicit reference to adaptation process may also be used for video processing when they are completed by temporal filtering steps. Filtering may be applied to the mapping function parameters. For example, Reinhard et al were inspired by dodging & burning photographic practices, and scaled HDR values to LDR ones according to the geometric averaged luminance (named « key value ») [START_REF] Reinhard | Photo-graphic tone reproduction for digital images[END_REF]. Temporal coherency is increased by filtering this key value over a number of frames which is fixed [START_REF] Kang | High dynamic range video[END_REF], or variable [START_REF] Ramsey | Adaptive temporal tone mapping[END_REF]. Alternately, filtering may be applied to the mapping function itself: in [START_REF] Ploumis | A Flickering Reduction Scheme for Tone Mapped HDR Video[END_REF] Ploumis et al proposed a method to differentiate between flickering effect and actual changes in real scene illumination. Whenever a flicker is detected, the Ward's mapping function is modified. When HDR luminance geometric mean variation appears to be due to a real change in scene characteristics, no filter is perform, and the frame is mapped according to the current conditions. Another approach to reach temporal coherency is post processing of tone mapped videos, as in [START_REF] Guthier | Flicker reduction in tone mapped high dynamic range video[END_REF]or [START_REF] Boitard | Zonal brightness coherency for video tone mapping[END_REF]. This particular domain is out of the scope of our work and will not be discussed any longer.

III. PROPOSED ALGORITHM

Logarithmic compression of luminance values is deemed to give a reasonably good fit of the human response to light. Our work is based on Drago's TMO [START_REF] Drago | Adaptive Logarithmic Mapping for Displaying High Contrast Scenes[END_REF], a global operator which offers a good compromise between performances and computational costs. It was one of the best graded techniques in the image subjective quality assessment conducted by [START_REF] Yoshida | Testing tone mapping operators with human-perceived reality[END_REF]. It also has the advantage to be easily adapted to suit the characteristics of the display. Many works already exist that are based on it, and logarithmic mapping is a research axe which is still being investigated. Aydin et al proposed to decompose HDR frames into base and details layers, et chose Drago's algorithm to scale HDR values of the base layer [START_REF] Aydin | Temporally coherent local tone mapping of HDR video[END_REF]. Popovic et al also used this algorithm for a real time implementation on a field programmable gate array [START_REF] Popovic | Performance optimization and FPGA implementation of real-time tone mapping[END_REF]. In [START_REF] Walbrecq | A Temporal Tone Mapping Adaptation: an Application using Logarithmic Camera[END_REF] the maximum luminance value of each frame was temporally filtered before being used in the tone mapping function. In a recent work, Lisani presented a local contrast enhancement based on logarithmic tone mapping curve [START_REF] Lisani | Local contrast enhancement based on adaptive logarithmic mappings[END_REF]. Pixels belonging to dark regions of the images are modified using a concave tone-mapping function and therefore their values are increased. Conversely, a convex tone-mapping is applied to pixels in bright regions, which produces a decrease of their values, and this results in an increase in visibility in both dark and bright regions.

A. Drago's TMO

This TMO relies on a logarithmic type tone curve applied on the scaled luminance Y channel in XYZ color space:

𝐿𝑤(𝑖, 𝑗) = ( , )   𝐿𝑤 =  
where L(i,j) is the HDR luminance value of pixel (i,j), which is equal to Y value in XYZ space, and Lmax is the maximum value in the HDR image. Lwavg is the image luminance geometric average, calculated from

𝐿𝑤 = 𝑒 ⋅ ∑ , , ( )

where  is a small constant used to avoid computational issue when L = 0. Lwavg is chosen as the scalefactor, as it commonly stands for visual adaptation state. In order to enhance the contrast, the base for the logarithmic function varies from 2 for the darkest pixels to 10 for the brightest ones, and smooth interpolation is performed between these values, based on a bias power function. The mapped values are further scaled to match the maximum luminance of the display 𝐿𝑑𝑚𝑎𝑥 before a conversion back to RGB space is performed:

𝐿𝑑 = ( ) ⋅ ( ) ⋅  with 𝑏𝑖𝑎𝑠 = ( ) ( . )



The b parameter is responsible for the trade-off between compression in dark and bright areas. A default value of 0.85 is suggested in [START_REF] Drago | Adaptive Logarithmic Mapping for Displaying High Contrast Scenes[END_REF].

B. Enhanced Drago TMO for Temporal Coherency (EDTC)

We propose to modify Drago's TMO with the objectives to avoid temporal artifact generation while ensuring a good overall contrast in all frames. Temporal coherency is obtained in a classical way, by filtering one of the different statistic parameters that describe each HDR frame. It was found that a leaky integrator applied to the maximum HDR value gives the best results:

𝐿 (𝑡) = 𝐿 (𝑡 -1) + 𝐿 (𝑡) -𝐿 (𝑡 -1) 𝑒 

The constant  is the ratio of the desired number of transition frames over the frame rate.

As already mentioned, contrast in the mapped images relies on the b value, it is then important that this value might be automatically adjusted to the overall luminance of the current frame: contrast in darker (resp. brighter) images is better for b near 0 (resp. 1). In consequence, b was set equal to the Ostu threshold for the histogram of Lw values. Ostu threshold is the value that separate histogram bins into two classes, with minimization of intra-class variance [START_REF] Otsu | A Treshold Selection Method from Gray-Level Histograms[END_REF]. In our case, the histogram is computed between the minimum and geometric mean values only. All values above this mean are added to its bin. The b value variations are further smoothed, according to the equation [START_REF] Ferwerda | A model of visual adaptation for realistic image synthesis[END_REF].

IV. EXPERIMENTAL ASSESSMENT

We evaluate the temporal performances of our algorithm in comparison with four state-of-the-art TMOs, in terms of temporal artifact generation.

A. Method 1) Definition of temporal artifacts

According to [START_REF] Boitard | Zonal brightness coherency for video tone mapping[END_REF], different categories of temporal artifacts may be defined. Temporal noise due to sensor defects is mainly noticeable in dark areas, and more disturbing than in static images, due to its high frequency characteristics. Because most of TMO increase the noise level rather than reducing it, a preliminary step of denoising is often necessary, in particular when videos are captured by HDR logarithmic sensor. Flicker occurs when the mapping function employs parameters deduced from image statistics which may undergo abrupt modifications from frame to frame, leading to transient changes in overall LDR frame brightness or incoherent pixel values. Temporal Brightness Incoherency (TBI) artifacts is observed when the overall relative brightness between two frames is different between HDR and LDR sequences. Temporal Object Incoherency (TOI) corresponds to the same phenomenon restricted to a limited zone or object in the scene. Color perception may also be perturbed by the mapping function, if one color channel gets clipped. Ghosting is another temporal artifact, which may arise on pixels located close to edges, especially for local TMOs, when temporal filtering is performed on a pixelwise basis. In our work, we mainly focus on flicker, TBI and TOI: as we only deal with global TMOs, ghosting is not likely to appear.

2) Others TMOs

We choose to compare our EDTC algorithm with four state-of-the-art global TMOs that are frequently assessed in comparison and review works. All these operators run in real time. The first one is the Photographic Tone Reproduction operator (Reinhard02) described in [START_REF] Reinhard | Photo-graphic tone reproduction for digital images[END_REF]. The second one, Kiser12, is another version of this TMO, with smooth filtering to reduce variation of the TMO parameters between adjacent frames [START_REF] Kiser | Real time automated tone mapping system for HDR video[END_REF]. Finally, Ferwerda96 [START_REF] Ferwerda | A model of visual adaptation for realistic image synthesis[END_REF] and Pattanaik00 [START_REF] Pattanaik | Timedependent visual adaptation for fast realistic image display[END_REF] are considered, as they perform rather well in [START_REF] Eilertsen | A comparative review of tone-mapping algorithms for high dynamic range video[END_REF] and [START_REF] Melo | Evaluation of HDR video tone mapping for mobile devices[END_REF]. TMO implementations were obtained from https://github.com/banterle/HDR_Toolbox. They were run with the same set of parameters for all videos, with the values obtained from a preliminary parameter adjustment experiment in [START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR Video[END_REF].

3) Video content

Depending on the scene content, performances may be different. We thus decided to use video sequences with a wide range of luminance variations. Some of these videos exhibit more or less rapid change in intensity, due to the camera moving in natural scenes (hallway, exhibition from [START_REF] Eilertsen | Evaluation of Tone Mapping Operators for HDR Video[END_REF], underbridgehigh from [START_REF] Boitard | Zonal brightness coherency for video tone mapping[END_REF] and window).

B. Results

The panels of Fig. 2 show the HDR mean luminance and LDR mean value over the different sequences. Flickering effect is demonstrated whenever the LDR mean value undergoes a sharp variation, while HDR mean value remaining merely constant. EDTC create videos totally free of such artifact, unlike Reinhard02, and Kiser12 (see in particular Underbridgehigh, Window, and Blinksource, in Fig. 2.) Temporal coherency may be disturbed, depending on the video under test. For relatively smooth luminance variations due to camera motion, as in hallway or window, only EDTC and Pattanaik00 perform well, the others showing strong temporal incoherencies, with LDR values undergoing variations decorrelated from the HDR ones. This behavior is even more pronounced for sharp variations, as in blinksource and lamp: in these cases, all but EDTC and Pattanaik00 lead to LDR mean variations opposite to HDR ones.

Temporal object coherency is also better preserved by these two algorithms. The temporal evolution of two pixels in lamp sequence (their location are indicated on Fig. 1) are shown on Fig. 3. For the pixel which becomes bright when the lamp is turned on (red point), brightness variations are coherent. For the point in the non-illuminated area (green point), the result is not so good. When the lamp is turned on, pixel value decreases. However, this incoherency is much less pronounced than for the others TMO.

Another characteristic of TMOs to be taken in account in their comparison is the way they manage luminance contrasts. The overall contrast relies on luma range in each frame, which is shown in Fig. 4. For the lamp sequence, when the lamp is turned on, the difference between the maximum and minimum values strongly decreases for all but EDTC TMOs, which reflects the fact that video appears less contrasted, with a kind of glare effect. In EDTC, dynamic range does not change, the feeling of contrast is maintained. This behavior is further highlighted in Fig. 5: when the lamp is turned on, some TMOs fail to render contrast in the bright area.

As expected, the Reinhard02 implementation we used, which was designed for image processing, shows the worst results. Its temporal adaptation in Kiser12, more surprisingly, performs only slightly better. Ferwerda96 is also prone to flicker and incoherencies. Only Pattanaik00 performs as well as EDTC concerning these artefacts. However, it produces quite saturated videos, as shown on Fig. 5.

CONCLUSION

A novel method for video tone mapping has been presented, which allows for real time processing. It has been tested on HDR sequences containing various luminance conditions, from smooth variations to sudden changes. In all cases, it produces artifact free videos, and maintains a good overall contrast.

Fig. 1 .

 1 Fig. 1. Three frames of the three video sequences captured for this work. Top row: lamp, with two measurement pixels indicated by green and red points; Middle row: blinksource; Bottom row: window.

Fig. 2 .

 2 Fig. 2. Temporal variations of the mean HDR luminance (left) and the mean LDR values (right) for the six sequences. a: hallway; b:exhibition; c:underbridgehigh; d:lamp; e:window; f:blinkingsource.

Fig. 3 .

 3 Fig. 3. HDR luminance (top row) and LDR values (bottom row) for two measurement points. Red pixel: in the lamp area. Green pixel: in the nonilluminated area (see Fig. 1 for details)


Olfa Haggui
email: olfa.haggui@mines-ales.fr

Hamza Bayd

Baptiste Magnier
email: baptiste.magnier@mines-ales.fr

Arezki Aberkane

Human Detection in Moving Fisheye Camera using an Improved YOLOv3 Framework

Keywords: Human detection, Fisheye camera, YOLOv3

Pedestrian detection has large relevance to the understanding of static and moving scenes of video sequences. The increasing demand for safety and security of people has resulted in more research on intelligent visual surveillance in a wide range of applications, such as moving human detection. With the great success of deep learning methods, researchers decided to switch from traditional methods based hand-crafted feature extractors to recent deep learning-based techniques in order to detect and track people. In this work, the topic of person detection with a Top-view moving fisheye camera is addressed. Although the fisheye camera is a useful tool for video monitoring, most of object detection techniques, with (or without) deep learning, concern classical perspective cameras. However, due to the distortions of fisheye images, we are expected to have higher requirements and challenges on the pedestrian detection using this device. In this paper, we propose an end-to-end learning people detection method based on YOLOv3 detector that detects people using oriented bounding boxes. The proposed model customizes the traditional YOLOv3 for the detection of oriented bounding boxes, by regressing the angle of each bounding box using a periodic loss function. With rotation bounding box prediction, our approach is efficient, reaching 98,1% of true detection. The proposed method is evaluated on a new available dataset where rotated bounding boxes represent annotations from several fisheye videos: https: //partage.imt.fr/index.php/s/nytmFqiq8jaztkX

I. INTRODUCTION

During the last few years, significant progress has been made in computer vision for people detection and tracking challenges, notably with the advancement of network technology. Within this context, typical cameras used in visual surveillance include perspective and fisheye cameras. Most of the existing researches are using perspective cameras, as they generate views similar to human vision, with small image distortions. However, its main disadvantages concerns the limited field of view. Therefore, the direct use of algorithms for classical cameras is not directly applicable on fisheye images.

People detection via video frames captured by fisheye cameras has received massive attention due to a certain number of advantages in visual surveillance application such as the large field of views. Yet, the major challenge is to take into consideration the radical distortions obtained in the image. Furthermore, the pedestrians in a fisheye image appear in different shapes, sizes and at various orientations, such as upright, upside-down, horizontal or diagonal. Unfortunately, most of the existing people-detection algorithms are designed for standard camera images where people appear upright. This paper focuses on the problem of people detection from video sequences recorded by Top-view moving fisheye cameras, as represented in Fig. 1, right. Over the past decade, a significant improvement has been witnessed with the help of traditional handcrafted features and models based on end-to-end learning. Among traditional people-detection algorithms, the most popular ones for pedestrian detection, HOG (Histogram of Oriented Gradients) and ACF (Aggregate Channel Features) have been used with overhead fisheye images. In [START_REF] Chiang | Human detection in fish-eye images using hog-based detectors over rotated windows[END_REF], the popular human detection algorithm based on the Histogram of HOG features and SVM (Support Vector Machines) classifier are combined after rotating each search window on a radial line to the vertical reference line. Another method in [START_REF] Wang | Template-based people detection using a single downward-viewing fisheye camera[END_REF] is based on HOG and LBP (Local Binary Patterns) features and SVM classifier to model people as upright cylinders and derived a series of elliptic detection masks whose size diminishes with the distance from the image center. In [START_REF] Demirkus | People detection in fish-eye top-views[END_REF], ACF are trained on side-view, standard-lens images for pedestrian detection without unwrapping a fisheye image into a panoramic image.

Recently, with the advent of Deep Learning, numerous benchmarks and datasets have been created in order to train and evaluate people detection algorithms with high accuracy in real-time. Some algorithms based on classification worked in two stages. First, the Regions of Interest (ROIs) are detected. This step represents a preprocessing, consisting of an image division into several regions using basic segmentations based on the colors or contours. Then, those regions are classified using Convolutional Neural Networks (CNN) or SVM. This process is very slow because every selected region must be predicted. In this context, the most popular algorithms are the Region-based convolutional neural network (RCNN) and the versions Fast-RCNN [START_REF] Lin | Pedestrian detection in fish-eye images using deep learning: Combine faster r-cnn with an effective cutting method[END_REF] and Faster-RCNN [START_REF] Ren | Faster r-cnn: Towards realtime object detection with region proposal networks[END_REF].

Instead of a selection of ROIs from the image, classes and bounding boxes (BBoxes) are predicted for the image in one run of the algorithm based on regression as in YOLO (You Only Look Once) [START_REF] Redmon | Yolov3: An incremental improvement[END_REF] and SSD (Single-Shot multibox Detection) mobilenet [START_REF] Yu | Omnipd: One-step person detection in top-view omnidirectional indoor scenes[END_REF] algorithms. Much research has addressed the topic of Top-view person detection with a static fisheye camera, mostly YOLO-based. In [START_REF] Tamura | Omnidirectional pedestrian detection by rotation invariant training[END_REF], a rotation invariant training method is applied, using randomly rotated standard images, without any additional annotation to simulate various poses and orientations of people in fisheye images. Another YOLO-based people detection method adapts YOLOv3 trained on standard images for people counting [START_REF] Li | Supervised people counting using an overhead fisheye camera[END_REF]. Each image is rotated in 15 • steps and YOLO is applied to the top-center part of the image followed by post-processing to remove multiple detections. Recently, the algorithm proposed in [START_REF] Duan | Rapid: rotation-aware people detection in overhead fisheye images[END_REF] provides much faster and more accurate results than previous algorithms aiming people detection in fisheye images, without any preprocessing. Its goal is to predict BBoxes of people, with certain center and size, but also the angle of each BBox.

Compared with the existing works, the technique presented in this paper can operate the detection of people in a complex scene recorded by Top-view moving fisheye cameras. No constraints on the peoples' movements is established, i.e., people can stand, sit, walk, kneel down, push objects and occlude each other for long periods of time. Moreover, this method does not require any camera calibration. To achieve this work, a new Top-view people detection dataset is introduced.

II. ORIENTED PEOPLE DETECTION VIA FISHEYE CAMERAS

A. Fisheye Camera Description

Usually, omnidirectional and fisheye cameras offer panoramic views of 2π radian angles [START_REF] Scaramuzza | Omnidirectional camera[END_REF]. Specific mirrors equip catadioptric cameras, whereas only lens concerns fisheye devices; then its angle of view can attain 2π radian angle or more. Therefore, objectives with wide-angle lens capture images typically warped, creating the effect of a fisheye. Fisheye cameras represent a major asset for several applications. In this way, these cameras are popular in many fields of computer vision, robotics and photogrammetric tasks such as navigation, localization, tracking, mapping and so on.

A fisheye camera is a camera fixed to a front lens group which appears as a single "big" lens, as shown in Fig. 1, left. This device enables a far greater negative refraction power than usual lenses, allowing greatly increasing the back focal distance and embracing wider fields of view [START_REF] Kumler | Fish-eye lens designs and their relative performance[END_REF]. In the context of people detection, the wide field of vision provided by these cameras makes people look inclined and distorted. Consequently, standard detection and tracking techniques are not reliable on warped images, especially with a cluttered and moving background [START_REF] Haggui | A comparison of opencv algorithms for human tracking with a moving perspective camera[END_REF]. Moreover, specific detectors for unconventional cameras are hard to design because they need a calibration stage which could be difficult to design [START_REF] Boui | New approach for human detection in spherical images[END_REF]. Even though many algorithms already exist for standard images, people detection and tracking regarding top-view images acquired by fisheye cameras are not a very documented topic and demand very specific involvement to work consistently.

B. Top View People Detection via Fisheye Cameras 1) Overview:

The focus here deals with real time people detection using a moving fisheye camera. There exist many methods for people detection and tracking using conventional camera, as referenced in [START_REF] Dollar | Pedestrian detection: An evaluation of the state of the art[END_REF]. However, people detection using fisheye cameras has been barely studied, due to the complexity of the device caused by the distortion effects. Additionally, in our investigation, we give extra focus on a people detection in a moving scenes in real-time. In recent methods, pedestrian detectors are trained using fisheye images, even though the manual labeling remains a hard task, which consumes times. Therefore, a new strategy for learning fisheye pedestrian detectors using images from a selected pedestrian dataset is proposed in this work. Another important development constraint, this detector should be equally applicable on a visual moving sensor that is either fixed in the environment or mounted on a mobile platform (like an aerial drone). To accommodate these challenges, a CNN model based on YOLOv3 detector is employed for person detection using top view video fisheye frames. This model is illustrated in Fig. 2. Its goal is to predict BBoxes of people, with certain center point position and size (width and height), but also the angle of each BBox. The angles of the BBoxes represent an important clue for the training or the detection. Indeed, rectangular BBoxes meet difficulties for object localization with different orientation angles, as produced by fisheye lens.

Actually, the proposed detector in this paper is a fully CNN with an architecture based on YOLOv3, and is configured to detect only one class, i.e., a person. In that respect, the network is structured in three parts. The first one represents the backbone network, known as Darknet-53, trained on the ImageNet database [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. Its main goal is to extract features at different spatial resolutions; it takes an input image and outputs a list of features from different parts of the network. Darknet-53 mainly consists of two blocks: residual and convolutional blocks. Each uses successive 1×1 and 3×3 convolution with doubly increasing filter channels, as well as shortcut connection between input and convolutional output, as summarized in Fig. 2. The second part concerns the Features Pyramid Network (FPN) [START_REF] Lin | Feature pyramid networks for object detection[END_REF]. This network takes as input the multiresolution features computed by our Draknet53 backbone in order to extract features related to person detection. In fact, FPN contains information about small and large objects. We expect D FPN 1 , the output of the FPN, to contain information about small objects and D FPN

3

, the output about large objects. The construction of this pyramid involves a bottom-up pathway, a top-down pathway, and lateral connections as shown in FPN block of the Fig. 2. The bottom-up pathway is the feed-forward computation of the backbone ConvNet, which computes a feature hierarchy consisting of feature maps at several scales with a down-scaling step of 2. The output of the last layer of each stage will be used as the reference set of feature maps for enriching the top-down pathway by lateral connection. For the top-down pathway, the higher resolution features are up-sampled by a factor of 2 spatially coarser, but semantically stronger, from higher feature maps of pyramid levels. These features are then merged with features from the bottom-up pathway via lateral connections. Finally, the third part is the head detection, allowing building a tensor T1,2,3 , containing information on the BBox position, including its angle of rotation. The implemented model uses a loss function combining Binary Cross Entropy (BCE , see Eqs. 2 and 3), as described in YOLOv3 [START_REF] Redmon | Yolov3: An incremental improvement[END_REF] [18], and, a periodic loss function that regresses the angle of each BBox, accounting for angle periodicities [START_REF] Ding | Learning roi transformer for oriented object detection in aerial images[END_REF] [10] [START_REF] Li | Supervised people counting using an overhead fisheye camera[END_REF]. Therefore, the detection of oriented objects is an extension of a general horizontal object detection.

2) Oriented Bounding Box Detection: In fisheye images, since most targets have an orientation, neither vertical, nor horizontal, rotated object detection is essential for overhead people detection (an example is available in Fig. 4). In our case, outputs of an improved YOLOv3 [START_REF] Redmon | Yolov3: An incremental improvement[END_REF] where b Conf is the predicted confidence score; it quantifies how confident the algorithm sounds that the target represents a human being. In addition, there is a confidence threshold, determined by the user, but usually fixed to 0.5, and the algorithm only returns the BBoxes whose confidence score is higher than this threshold. The Fig. 3 shows the transform from the anchor to the BBox where the coordinates center (b x , b y ) of BBox is calculated by applying a sigmoid to predicted values and adding the corner points of the corresponding grid cell. Meanwhile, the dimensions b w and b h of the BBox are calculated by applying a log-space transform to the predicted output dimensions and then multiplying with an anchor dimensions (p w , p h ). The network establishes a multitask and crucial loss function (Eq. 1) inspired by that used in YOLOv3, with an additional BBox rotation-angle loss to optimize the target detection. It is computed by the ground truth and the predicted result of the network:

Loss = Loss Box + Loss Conf + Loss Angle , (1)
where the Box regression loss (Loss Box in Eq. 2) is calculated only when the prediction box contains detected people.

Confidence loss (Loss Conf in Eq. 3) determines whether there are persons in the prediction frame. BBox rotationangle loss (Loss Angle in Eq. 4) determines the prediction orientation of a person. Note, that the category-classification loss is not used since only one class (i.e., persons) is used here.

Here, T = ( tx , ty , tw , th , tθ , tconf ) represents the transformed

+ t∈ T + S( tw) -tw 2 + S( th ) -t h 2 , ( 2
)
Loss Conf = t∈ T + BCE S( tconf ), 1 + t∈ T - BCE S( tconf ), 0 , (3)
with S the logistic sigmoid activation function, ( T + , T -) positive and negative samples from the predictions respectively.

Loss Angle = t∈ T + L Angle bθ , b θ , (4)
such that for a given range (α, β), bθ = α • S( tθ ) -β, which is the predict of b θ and the function L Angle is defined by

L Angle bθ , b θ = R mod bθ -b θ - π 2 , π - π 2 ,
with "mod" representing the modulo operation and R a symmetric regression function.

III. IMPLEMENTATION DETAILS A. Dataset Description

Numerous benchmarks and datasets have been created in order to train and evaluate people detection algorithms regarding fisheye images. Most of the existing public fisheye datasets are annotated by an aligned BBox. In this work, a dataset of overhead fisheye images with oriented BBoxes is needed for each person aligned with its orientation in the image. However, different challenges are reported with the dealing of fisheye images, mainly spatial and temporal illumination variations, occlusions, and various body poses for example. Additionally, when people are walking straight under the camera and at the periphery of the fisheye image, the appearance is different and the image resolution is small near the borders, disturbing regularly the detection. To overcome these challenging scenarios of different videos captured from a moving fisheye camera, a new dataset has been collected and Fisheye cameras (OBBMF) and is composed of 6 videos (see Tab. I for details). Clearly, the new dataset contains many more frames and human objects, and also includes challenging scenarios, which do not exist in the other datasets. Furthermore, experiments are performed using three public datasets, MW-18Mar1 , HABBOF 2 , CEPDEOF 3 , and our datasets in order to fit and evaluate the effectiveness of our proposed method.

B. Data Acquisition

In our case, the data are collected in the CERIS laboratory of the IMT Alès research center. The videos were collected with fisheye camera (Basler ace acA1300-200uc) facing down at 48 fps, where one or several persons perform under the camera various poses such as walking and sitting. Also severe body occlusions are present, and people go up and down the stairs with rotational movement. Then, a number of frames are generated from these video clips which contains new scenes with some challenging scenarios such as illumination, camera rotation and motion in the center (Top-view), which are generally unavailable in the common literature

C. Data Annotation and File Conversion

Data labelling is an essential step in a supervised machine learning task requiring a lot of manual work. To annotate our new dataset, person regions are manually annotated with rotated BBoxes. In order to do so, the MVTec Deep Learning Tool 4 is used. It is a very useful target detection and labeling deep learning tool in a short time, but it generates a hdict file which cannot be directly used in other deep learning tools. So, it is converted into txt file in the first step using a small application with C# and HalconDotNet-V. 19.11.0, and finally converted again to the data format of YOLOv3 as a json file. Technically, first, the main axis of the rectangle is drawn to define the orientation of the person in the scene. Then the width and height of the person in pixels are defined. Consequently, each BBox is represented by five parameters:

• (x, y): coordinates of the BBox center,

• w and h: width and height of the BBox respectively,

• θ: clock-wise rotation angle from the vertical axis. Furthermore, Figs. 4(a) and (b) represent these parameters in a rectangle BBox. The whole of our data set is represented by more than 12.000 OBBMF. However, some annotations in the MW database contain errors in the angles, as shown in Fig. 4(b). Consequently, these annotations have been manually modified to improve the relevance of this database. It allows bringing a very efficient model for human being detection.

D. Training Datasets

Concerning training, a pre-trained Darknet53 model was used as a starting point, which is initialized with ImageNet pre-trained weights for faster training. Also, in order to train the proposed detector, we used one of the largest dataset MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] that is commonly utilized as a general object detection benchmark, because it contains various appearances of people. Our network was trained end to end by optimizing the cross entropy loss function by updating weights using "Stochastic Gradient Descent" (SGD) [START_REF] Ramezani-Kebrya | On the generalization of stochastic gradient descent with momentum[END_REF] with a momentum of 0.9 for more than 50,000 iterations (one iteration contains 128 images). The SGD represents an iterative optimization technique [START_REF] Bottou | Large-scale machine learning with stochastic gradient descent[END_REF], and is most widely used in the field of deep learning to minimize the loss function to search hyperparameters. This algorithm calculates the gradient and makes the update of the network parameters by the mean of the training set'subset, which is called mini-batch. Each gradient evaluation using the mini-batch is defined as an iteration. At each iteration, the algorithm takes one step to minimize the loss function. The complete progress of the training algorithm over the total training set using mini-batches is called an epoch. During experiments, the initial learning rate is set to 0.001 and the weight decays to 0.0005. The mini-batch size is set to 16, and the network is trained for 500 epochs. We have set the learning rate factor to 0.0001 with the same SGD parameters and batch size to fine-tune parameters of the network. All the process was trained on multiple cross fisheye datasets for more than 8000 iterations from weights pre-trained in ImageNet using COCO. For these two networks, the images were resized into 608×608 pixels and fed into the network until the loss has been saturated. Our model was conducted on an NVIDIA Quadro P5000 GPU accelerator (Pascal architecture). It includes 2560 CUDA cores with 16 GB GDDR5 memory. The host is an Intel® Xeon® CPU E5-1620 V4 processors with 4 cores.

IV. EXPERIMENTAL RESULTS AND EVALUATIONS A. Evaluation Metrics

The detection system returns a list of detected BBoxes in an image. The match of a detected BBox and the ground truth is rated by asserting an overlap area of more than 50%. To quantitatively evaluate the performance of the proposed network, the statistical analysis of P recision, Recall, Fscore and Average Precision (AP ) are performed as the evaluation metrics. For T P , F P and F N denoting le number of true positives, false positives and false negatives in a video, P recision means the percentage of the correctly detected persons (T P ) over all the detected persons (T P + F P ): P recision = T P T P +F P . Meanwhile, Recall is the ability of a model to find all the objects. It associates with the correct predictions among all the positive cases, which means the percentage of the correctly detected: Recall = T P T P +F N . Hense, P recision and Recall are considered to be common evaluation metrics, and the F -score combines the two:

F = 2 • P recision × Recall P recision + Recall . (5)
Finally, Average Precision (AP ) is the area under the Precision-Recall curve: AP = 1 0 F (x) dx. Therefore, the closer the evaluation scores of both F and AP are to 1, the more the detection is qualified as suitable. On the contrary, a score close to 0 corresponds to a poor detection of persons.

B. Benchmark Results

Various experiments are presented to analyze the performance of the proposed method. We fine-tuned the algorithm trained on COCO with various cross datasets from MW-R, CEPDEOF, HABBOF and OBBMF. Hence, we cross-validate on these datasets, i.e., two datasets are used for training, then they are tested on another dataset. For example, a cross dataset trained on MW-R + HABBOF is tested on OBBMF, and inversely. Tab. II shows the detection performance of our method on each video in the OBBMF dataset obtained by fine-tuning with cross-validation1 (index cross1) and cross-validation2 (index cross2), respectively. Our model with 608×608 resolution achieves impressive performance, despite of using videos captured from moving fisheye camera. Thus, the proposed method performs outstandingly with an acceptable convergence behaviours in several experiments carried out with various cross validation datasets.

The overall performance metrics obtained with our model and with our OBBMF dataset are evaluated using AP , P recision, Recall and F -measure. As shown in the Tab. II, the new algorithm performs efficiently on ordinary videos with a score more than 0.95 for AP . However, more complex scenes (moving camera, low light, strong shadows, etc.) remain challenging. The Fig. 6 shows sample results applied to the four datasets where detections are nearly perfect in a range of scenarios, such as various body poses, orientations, and diverse background scenes. For this reason, we expand our training datasets by cross validating various samples from all the used datasets, in order to improve the resulting performance model and prevent it from over-fitting. The resulting cross dataset was split: 70% used for the training stage and 30% from data are used for testing. On the one hand, the Tab. III shows clearly the improved performance with large scales datasets, exceeding 90% of the AP . On the other hand, the P -R (P recision-Recall) curve of the best model on the test set is plotted in the Fig. 5; it shows the trade-off between the P recision and the Recall as the threshold score of the model changes. Recall should increase to guarantee that all the persons are detected. However, as Recall increases, it is common for some scenarios, such as distortion, camera movements, low light, and strong shadows, decreasing P recision. Ideally, the upper-right corner of the curve should reflect 100% of Recall and P recision, often impossible to obtain in real scenarios. Eventually, P -R curve illustrates why the AP of our model is high. Indeed, P recision remains close to 100% for Recall values as high as 85%, with a 0.98% for the optimal point. V. CONCLUSION An approach is proposed in this paper to detect people in Top-view fisheye images, using moving camera. It is based on a pre-trained deep CNN architecture, extended from YOLOv3 detector. Experimental results confirm that people are extracted in an indoor environment using videos streaming from a fisheye moving camera with a high level of AP . Our approach eliminates the need for pre-processing and/or data augmentation, by considering oriented bounding boxes. Finally, a new dataset of videos has been created regarding human detection with a top view fisheye moving camera; the great interest is that the ground truths are also available online.

For future works, we plan to expand the proposed model for more large and complex datasets at different image resolutions. We believe both that our method and dataset will be beneficial for various real-world applications, especially for human detection and tracking using overhead fisheye videos captured and treated automatically from an aerial drone.

Camera

Fig. 1 .

 1 Fig. 1. Fisheye camera. On the left: fisheye lens used in our experiments: 2/3" Format C-Mount fisheye lens 1.8mm FL, with a horizontal field of view, 1/2" sensor for 185°. On the right: diagram of the experimental protocol.

T 3 T 2 T 1 Fig. 2 .

 3212 Fig. 2. Architecture of the proposed network. input: fisheye image, backbone (Darknet53), Features Pyramid Network (FPN), and detection head (BBox regression network), Oriented BBoxes as outputs. For the YOLO loss function, only one class, 5 parameters for each BBox and a confidence score.

  network are used with both horizontal location boxes and angle information, rendering YOLOv3 module more sensitive to the angle. By introducing the oriented BBoxes, for each video frame, the predicted results of the proposed framework return six BBox parameters: the position coordinates (b x , b y ), the BBox size (b w , b h ) and the angle of all individuals b θ . They are represented by a six-dimensional vector (b x , b y , b w , b h , b θ , b Conf ),

Fig. 3 .

 3 Fig. 3. Oriented Bounding Box (BBox) with its tied parameters.

Fig. 4 .

 4 Fig. 4. Examples of annotated frames with BBox rotation angle. In (b), a frame and its BBoxes from MW-R dataset (the red BBoxes are the initial tied to the MW-R dataset whereas the green correspond to the corrected).

Fig. 5 .

 5 Fig. 5. Precision-recall curve of the best model on the test set. The precision remains close to 100 % for recall values as high as 85%. The optimal point (the closer to the upper-right corner) is at 0.981.

Fig. 6 .

 6 Fig.6. Detection results of our benchmark on sample frames in different scenarios and challenge, including various poses, orientations and background scenes. Green boxes are predicted BBox (true positives, i.e., matching of a detected BBox and the groundtruth with an overlap area of more than 50%).

TABLE I DESCRIPTIONS

 I OF VIDEO SEQUENCES AND THEIR TIED CHALLENGES annotated. It is called Oriented Bounding Boxes from Moving

TABLE II PERFORMANCE

 II COMPARISON OF OUR METHOD WITH CROSS VALIDATION cross1 AND cross2 FOR EACH VIDEO IN OBBMF DATASET.

TABLE III PERFORMANCE

 III TUNING EVALUATION OF OUR METHOD WITH MULTI CROSS VALIDATION FOR EACH VIDEO IN EACH DATASET.

http://www2.icat.vt.edu/mirrorworlds/