Mohamed Kerkech

Adel Hafiane

Raphael Canals

Vine disease detection in UAV multispectral images using optimized image registration and deep learning segmentation approach

Keywords: Unmanned aerial vehicle (UAV), image registration, convolutional neural network, precision agriculture, disease mapping

One of the major goals of tomorrow's agriculture is to increase agricultural productivity but above all the quality of production while significantly reducing the use of inputs. Meeting this goal is a real scientific and technological challenge.

Smart farming is among the promising approaches that can lead to interesting solutions for vineyard management and reduce the environmental impact.

Automatic vine disease detection can increase efficiency and flexibility in managing vineyard crops, while reducing the chemical inputs. This is needed today more than ever, as the use of pesticides is coming under increasing scrutiny and control. The goal is to map diseased areas in the vineyard for fast and precise treatment, thus guaranteeing the maintenance of a healthy state of the vine which is very important for yield management. To tackle this problem, a method is proposed here for Mildew disease detection in vine field using a deep learning segmentation approach on Unmanned Aerial Vehicle (UAV) images. The method is based on the combination of the visible and infrared images obtained from two different sensors. A new image registration method was developed to align visible and infrared images, enabling fusion of the information from the two sensors. A fully convolutional neural network approach uses this information to classify each pixel according to different instances, namely, shadow, ground, healthy and symptom. The proposed method achieved more than 92% of detection at grapevine-level and 87% at leaf level, showing promising perspectives

Introduction

Several studies have been carried out on the overuse of crop pesticides and their negative effects on human health [START_REF] Aktar | Impact of pesticides use in agriculture: Their benefits and hazards[END_REF][START_REF] Pimentel | Environmental and economic effects of reducing pesticide use in agriculture[END_REF][START_REF] Hernández | Pesticide abuse in Europe: Effects on the Cinereous vulture (Aegypius monachus) population in Spain[END_REF]. Like other crops, vines are very vulnerable to viruses, bacteria and fungi. This vulnerability favours their contamination by several types of disease that are harmful and destructive [START_REF] Tyerman | Water transport & aquaporins in grapevine[END_REF], such as Esca [START_REF] Hofstetter | What if esca disease of grapevine were not a fungal disease?[END_REF], Flavescence dore [START_REF] Chuche | Biology and ecology of the Flavescence dorée vector Scaphoideus titanus: A review[END_REF] and Mildew [START_REF] Gessler | Plasmopara viticola: A review of knowledge on downy mildew of grapevine and effective disease management[END_REF]. Vine contamination generally reduces productivity [START_REF] Macdonald | Remote hyperspectral imaging of grapevine leafroll-associated virus 3 in cabernet sauvignon vineyards[END_REF], which implies economic losses for the winegrower.

To deal with this situation, winegrowers have to frequently check the state of the vine leaves. However, this traditional procedure is laborious and expensive, since it involves several experts for many days [START_REF] Al-Saddik | Damav : Un Projet Interregional De Detection De Foyers Infectieux De Flavescence Doree Par Imagerie De Drone, Journée technique VITINNOV Viticulture de précision : les capteurs à la loupe[END_REF]. To reduce economic loss and the environmental impact, remote sensing methods are a promising approach for effective vineyard monitoring.

Remote sensing of agricultural crops [START_REF] Teke | A short survey of hyperspectral remote sensing applications in agriculture[END_REF] has evolved considerably over the past decade. Applications such as calculating fertilizer rates [START_REF] Schut | Assessing yield and fertilizer response in heterogeneous smallholder fields with UAVs and satellites[END_REF], monitoring biomass production [START_REF] Karpina | Uavbased automatic tree growth measurement for biomass estimation[END_REF], weed detection [START_REF] Bah | Weeds detection in UAV imagery using SLIC and the hough transform[END_REF], detecting defective crops [START_REF] Altnba | Detecting Defected Crops: Precision Agriculture Using Haar Classifiers and UAV[END_REF] or disease detection [START_REF] Khirade | Plant disease detection using image processing[END_REF][START_REF] Pinto | Crop disease classification using texture analysis[END_REF][START_REF] Schor | Robotic Disease Detection in Greenhouses: Combined Detection of Powdery Mildew and Tomato Spotted Wilt Virus[END_REF] have been proposed. These applications are constantly progressing as technology advances, especially with the evolution of Unmanned Aerial Vehicles (UAV) which have opened up further research opportunities thanks to their low manufacturing costs.

UAVs are increasingly used in many fields, such as urban remote sensing, but also in a wide range of agricultural applications [START_REF] Barbedo | A Review on the Use of Unmanned Aerial Vehicles and Imaging Sensors for Monitoring and Assessing Plant Stresses[END_REF]. Previous studies have shown the importance of both the visible and the infrared spectrum for disease detection [START_REF] Zhu | Application of infrared thermal imaging for the rapid diagnosis of crop disease[END_REF]. Combining these two imaging modalities would therefore ensure better detection. In UAV imaging systems, usually two separate sensors are used, one for each modality. However, an acquisition by two sensors generates a spatial shift between the visible and infrared image which makes it difficult to process the information from the two sensors simultaneously. Therefore, multimodal alignment or registration [START_REF] Peñaranda | Multimodal registration of optical microscopic and infrared spectroscopic images from different tissue sections: An application to colon cancer[END_REF][START_REF] Wang | A deep learning framework for remote sensing image registration[END_REF][START_REF] Tsai | An accelerated image matching technique for UAV orthoimage registration[END_REF][START_REF] Inostroza | Embedded registration of visible and infrared images in real time for noninvasive skin cancer screening[END_REF] is required to merge the information from the two sensors with deep learning.

Deep learning techniques have enabled great progress in the computer vision field thanks to the convolutional neural networks (CNNs) approach [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF][START_REF] Monien | Approximation algorithms for multilevel graph partitioning[END_REF][START_REF] Zeiler | Visualizing and Understanding Convolutional Networks BT -Computer VisionECCV[END_REF][START_REF] Zhang | An empirical-theoretical calculation method and fullscale verification test for the ultimate bearing capacity of an expanded pedestal uplift pile[END_REF][START_REF] Zeng | Inception-Net/GoogLeNet -Going Deeper with Convolutions[END_REF][START_REF] He | Deep residual learning for image recognition[END_REF]. As in several fields of application, these technologies are increasingly used in the remote sensing domain for agriculture [START_REF] Kamilaris | Deep learning in agriculture: A survey[END_REF][START_REF] Chen | Deep learning techniques for agronomy applications[END_REF][START_REF] Sladojevic | Deep Neural Networks Based Recognition of Plant Diseases by Leaf Image Classification[END_REF][START_REF] Lu | Identification of rice diseases using deep convolutional neural networks[END_REF][START_REF] Yang | Deep convolutional neural networks for rice grain yield estimation at the ripening stage using UAV-based remotely sensed images[END_REF][START_REF] Fuentes | A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition[END_REF][START_REF] Lee | A self-predictable crop yield platform (SCYP) based on crop diseases using deep learning[END_REF][START_REF] Bah | Deep learning based classification system for identifying weeds using high-resolution UAV imagery[END_REF][START_REF] Kerkech | Deep learning approach with colorimetric spaces and vegetation indices for vine diseases detection in UAV images[END_REF]. Most research on agricultural applications uses CNNs with the sliding window technique, which generally leads to fuzzy boundaries of the image regions.

On the other hand, crop disease detection can be seen as an image segmentation problem. Therefore, one can benefit from the deep learning segmentation approach to detect crop disease with a better boundary precision compared to the sliding window technique. Several segmentation architectures have been developed, such as SegNet [START_REF] Badrinarayanan | SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation[END_REF], DeconvNet [START_REF] Noh | Learning deconvolution network for semantic segmentation[END_REF] and U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. SegNet is the most popular architecture for semantic segmentation [START_REF] Kendall | Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene UnderstandingarXiv[END_REF][START_REF] Alonso | Semantic Segmentation for Eventbased CamerasarXiv[END_REF][START_REF] Audebert | Segment-before-detect: Vehicle detection and classification through semantic segmentation of aerial images[END_REF][START_REF] Guo | Dark spot detection in SAR images of oil spill using segnet[END_REF]. It has shown a very good performance in solving problems related to semantic segmentation for several applications [START_REF] Nguyen | Hand Segmentation and Fingertip Tracking from Depth Camera Images Using Deep Convolutional Neural Network and Multi-task SegNetarXiv[END_REF][START_REF] Du | Training SegNet for Cropland Classification of High Resolution Remote Sensing Images[END_REF][START_REF] Wei | Autonomous Navigation of an Indoor Mecanum-Wheeled Omnidirectional Robot Using Segnet[END_REF]. So far, very little attention has been paid to the role of deep learning segmentation for vine disease detection. This paper presents a new methodology for Mildew disease detection in vineyards from aerial images, using multispectral information. The aim was to develop algorithms and methods in order to investigate the possibility of detecting vine diseases, using the potential of a deep learning segmentation architecture. The problem was addressed by the semantic segmentation approach in order to identify classes such as shadow, ground, healthy and symptomatic vines. The method consists in two main steps. The first one deals with the problem of multispectral image registration, and proposes a new method to effectively align images from the visible and infrared spectra. The second one uses the SegNet architecture to delineate semantic areas in each image type separately, then a fusion procedure was applied to the segmentation outputs.

Data were collected under real conditions on two vineyard plots. Several experimental schemes were set up to show the contribution of different elements of the proposed method. The study provides important insight into the potential of recent machine learning approaches for disease mapping using UAV remote sensing technology.

The main contributions of this work are: optimization of the image registration algorithm, introduction of a semi-automatic method for pixel-wise image labelling and SegNet for vine disease detection in the visible and infrared ranges.

The article is organized as follows: related work is presented in section 2, the study areas and materials are described in section 3, the proposed methods are detailed in section 4, the experiments and results are presented in section 5, the proposed system is discussed in section 6 and section 7 concludes the paper.

Related work

This section summarizes the main studies carried out on image registration, and disease detection in vineyards, plants or crops.

Image registration

In the literature, the problem of image registration dates back to the 1980s. Since then, several methods have been implemented. The work accomplished in various fields has been surveyed in: medicine [START_REF] Ferrante | Slice-to-volume medical image registration: A survey[END_REF], computer vision [START_REF] Zitová | Image registration methods: A survey[END_REF], remote sensing [START_REF] Dawn | I. Technology, Remote Sensing Image Registration Techniques : Survey, Image and Signal Processing[END_REF] and various applications [START_REF] Nag | Image Registration Techniques: A SurveyarXiv[END_REF]. In all areas studied, it is concluded that image registration is based on two main methods: the area-based method, and the feature-based method.

The area-based method: This method is not widely used in the remote sensing field, because most of the algorithms are highly sensitive to several uncontrollable parameters, such as variations in brightness, image noise, etc. The method is therefore generally applicable only to non-rigid problems. However, Wang et al. [START_REF] Wang | Efficient registration of optical and IR images for automatic plant water stress assessment[END_REF] implemented an algorithm (An automatic cross-correlation (ACC)) insensitive to the lighting conditions and applied it to multimodal images (visible and infrared). The authors concluded that this algorithm performs better than other area-based algorithms and is suitable for multimodal images. Another registration method applied to precision agriculture was proposed by Erives et al. [START_REF] Erives | Automated registration of hyperspectral images for precision agriculture[END_REF]. This method that processes multispectral images is based on the phase correlation algorithm (PC). The results obtained indicate that this algorithm is robust to modality change, brightness difference, noise, rotation and translation. Zhuang et al. [START_REF] Zhuang | Infrared and visual image registration based on mutual information with a combined particle swarm optimization -Powell search algorithm[END_REF] performed a multimodal registration based on mutual information with a combination of Particle Swarm Optimization (PSO)

and Powell search algorithms. The proposed method was found to be faster in terms of runtime and more accurate in terms of results compared to traditional methods.

The feature-based method: Lakshmi et al. [START_REF] Sreenu | Control of Brushless DC Motor with Direct Torque and Indirect Flux using SVPWM Technique[END_REF] and Javadi et al. [START_REF] Javadi | An Automatic Robust Image Registration Algorithm for Aerial Mapping[END_REF] worked on natural terrain and city video frames, acquired by a UAV for the creation of an orthorectified image. In these studies, the standard registration method based on the SURF algorithm was used. In [START_REF] Sreenu | Control of Brushless DC Motor with Direct Torque and Indirect Flux using SVPWM Technique[END_REF], this method was compared with the Cross-Correlation algorithm (Area-based method) which failed to register the aerial images, whereas the feature-based method using the variants of the SIFT algorithm outperformed the other methods in terms of results and in terms of runtime. Tsai et al. [START_REF] Tsai | An accelerated image matching technique for UAV orthoimage registration[END_REF] conducted a similar study to compare the SIFT algorithm with ABRISK, and concluded that ABRISK was up to 312 times faster than SIFT, and had a lower mean error. In another study, Onyango et al. [START_REF] Onyango | Accurate estimation of orientation parameters of UAV images through image registration with aerial oblique imagery[END_REF] used the AKAZE algorithm to match oblique building images with images of cities taken by a UAV. The study concluded that the AKAZE algorithm outperformed other algorithms of the same type. More recently, image matching algorithms based on deep learning have appeared [START_REF] Yang | Multi-temporal remote sensing image registration using deep convolutional features[END_REF][START_REF] Wang | A deep learning framework for remote sensing image registration[END_REF]. The deep learning architecture is used as a feature extractor to create a correspondence between the two images. Wang et al. [START_REF] Wang | A deep learning framework for remote sensing image registration[END_REF] worked on remote sensing images using a supervised architecture, while Yang et al. [START_REF] Yang | Multi-temporal remote sensing image registration using deep convolutional features[END_REF] used an unsupervised architecture to recalibrate multi-temporal images. The latter showed better accuracy than the SIFT type algorithms, but these results only correspond to multi-temporal images, of the same modality and on low resolution images.

Disease detection

A comprehensive review of the literature was conducted by Mahlein [START_REF] Mahlein | Plant disease detection by imaging sensors Parallels and specific demands for precision agriculture and plant phenotyping[END_REF].

The survey lists several studies on disease detection by multispectral imaging.

Among others, Oerke et al. [START_REF] Oerke | Thermographic assessment of scab disease on apple leaves[END_REF] showed by using a thermal camera that the leaves of plants in the early stages of a disease have a more heterogeneous temperature than when the disease progresses over time. The potential of multispectral information in the early detection of plant disease is attracting more and more interest in the remote sensing field. Two studies on the hyperspectral reflectance of vine leaves diseased by the Esca complex have been carried out. The first one [START_REF] Junges | Detection of grapevine leaf stripe disease symptoms by hyperspectral sensor[END_REF] was performed at the leaf level and the second one [START_REF] Di Gennaro | Unmanned Aerial Vehicle (UAV)-based remote sensing to monitor grapevine leaf stripe disease within a vineyard affected by esca complex[END_REF] at the vineyard level (UAV images). Both showed a difference between the reflectance of a healthy and a diseased leaf.

In a first study, Albetis et al. [START_REF] Albetis | Detection of Flavescence dorée grapevine disease using Unmanned Aerial Vehicle (UAV) multispectral imagery[END_REF] investigated the detection of Flavescence dore in UAV images. The study was carried out on plots of white and red cultivars. The results obtained indicate the feasibility of disease detection using aerial images. In a second study, Albetis et al. [START_REF] Albetis | On the potentiality of UAV multispectral imagery to detect Flavescence dorée and Grapevine Trunk Diseases[END_REF] examined the potential of multispectral imaging by UAV for the detection of symptomatic and asymptomatic vines. In addition to the first study, a larger dataset was acquired and used to test 24 variables calculated from this new dataset. The best results were obtained by the red-green index (RGI) and the red-green vegetation index (GRVI).

In Al-Saddik et al. [START_REF] Al-Saddik | Development of spectral disease indices for flavescence dorée' grapevine disease identification[END_REF], a first study on hyperspectral images at the vine leaf scale was carried out. The aim of the study was to develop spectral disease indices capable of detecting and identifying Flavescence dore disease in vines, and achieved 90% classification accuracy. A second study by Al-Saddik et al. [START_REF] Al-Saddik | Using image texture and spectral reflectance analysis to detect Yellowness and Esca in grapevines at leaf-level[END_REF] on disease identification at the vine leaf level was carried out to differentiate between yellowing leaves, and leaves infected with Esca disease through a neural network classifier. The best results were obtained when textural and spectral data were combined. A third study by the same authors [START_REF] Al-Saddik | Assessment of the optimal spectral bands for designing a sensor for vineyard disease detection : the case of Flavescence dorée[END_REF] consisted in defining the most significant spectral channels for Flavescence dore disease detection.

Ranon et al. [START_REF] Rançon | Comparison of SIFT encoded and deep learning features for the classification and detection of esca disease in Bordeaux vineyards[END_REF] carried out a similar study on Esca disease detection in vineyards. The imaging system was mounted on a small vehicle that passed between the vinerows for image acquisition. Two methods for detecting Esca disease were used: Scale Invariant Feature Transform (SIFT) encoding, and the deep learning MobileNet network. The authors concluded that the deep learning method was better than the SIFT encoded method.

In our previous study [START_REF] Kerkech | Deep learning approach with colorimetric spaces and vegetation indices for vine diseases detection in UAV images[END_REF], a new method for detecting Esca disease in UAV RGB images was proposed. The method used LeNet5 CNN architecture and good results were obtained achieving 95% disease detection accuracy in the visible range.

To the best of our knowledge, no research has been conducted so far on the combination of visible and infrared UAV images for vine disease detection by a deep learning segmentation approach.

Study areas and materials

Study areas

This study was carried out on two parcels of vines located in the Center Val de Loire region in France. The first plot (P1) can be seen in Figure 1a and the second one (P2) in Figure 1b. P1 and P2 are at an altitude of 110 and 114 meters respectively, with a surface of 1.8 hectares for P1 and 1.5 hectares for P2, and are positioned on a silty sand soil for P1 and sandy loam soil for P2. The ground of P1 is slightly inclined (7% of slope) (Figure 2) and flat for P2. The Centre Val de Loire region is characterized by a moderate temperature variation between 1 to 26 • C, and the average annual rainfall reaches 700 mm. Table 1 gives more details about the plots.

In order to carry out this study and obtain healthy and diseased samples, part of the P1 plot was treated with phytosanitary products (to protect the vine against diseases), and the other part remained untreated in order to allow the development of the disease. During this time, Mildew disease spread to all untreated areas of the plot. The P2 plot was treated globally against disease and remained totally healthy. It was used in this study for qualitative validation.

Materials

The UAV used in the data acquisition process was a Quadcopter drone (Figure 3a) manufactured by Scanopy. This drone embeds two camera sensors MAPIR Survey2 (Figure 3b). The first is a visible light sensor (RGB) set to automatic lighting, and the second is an infrared light sensor (Near Infrared (NIR), Red and Normalized Difference Vegetation Index (NDVI)). For the latter, the near infrared wavelength is 850 nm. Both image sensors have a high-resolution of 16 megapixels (a size of 4608 × 3456).

Data acquisition was carried out by the UAV, which flew over the plot at an altitude of 25 meters and with an average speed of 10 km/h. At this altitude, the ground resolution is 1cm 2 /pixel. An image was taken every 2 seconds without stopping the UAV. Each image taken overlaps the previous image by

Methods

Using the multispectral and standard RGB images, automatic processing and analysis methods were developed to extract relevant information and correlate it with the ground truth results. Deep learning segmentation was applied on the two types of images to automatically delineate different regions (healthy, symptomatic, etc.). This generated a disease map of the vineyard comprising different segmented regions, which was used to monitor the vineyards.

The method comprised three main steps (see Figure 4). The first one consisted in image registration between the images acquired in the visible and infrared range. This step is essential for the third step, as it enables the pixel-wise superposition of the two images and thus allows segmentation fusion. Once the registration of the two images had been performed, the next stage consisted in segmenting the plot in the visible and infrared range using the SegNet architecture. The two segmented images were merged in the third step, to generate a disease map.

Image registration

The objective of the registration algorithm is to realign and geometrically correct the shift [START_REF] Szeliski | Image Alignment and Stitching: A Tutorial[END_REF] between the visible and infrared images at the pixel-wise level. Generally, the UAV image is partially distorted due to the UAV vibrations, read in rolling shutter mode of the sensors and optics. Therefore, the rigid alignment model (translation and rotation) is not appropriate in our case.

Hence, the proposed algorithm used the non-rigid model. However, even if the registration model is correct it is not sufficient to align the two types of images perfectly. Moreover, matching image points between the visible and infrared bands is difficult, since the key points do not necessarily have the same visual properties in the two spectral bands. In order to improve the accuracy of the image alignment, an iterative process based on minimization of the registration error was implemented.

The registration algorithm proposed in this study is based on the Accelerated-KAZE (AKAZE) algorithm [START_REF] Alcantarilla | Fast explicit diffusion for accelerated features in nonlinear scale spaces[END_REF]. AKAZE is an algorithm used in computer vision for detecting objects or similarities in two images. Its principle is comparable to that of the Scale Invariant Feature Transform (SIFT) [START_REF] Lowe | Distinctive Image Features from Scale-Invariant Keypoints[END_REF], Speeded Up Robust Features (SURF) [73], Features from Accelerated Segment Test (FAST), Binary Robust Independent Elementary Features (BRIEF), Oriented FAST and Rotated BRIEF (ORB) [START_REF] Hasenbusch | The critical behavior of 3D Ising spin glass models: Universality and scaling corrections[END_REF], and KAZE [START_REF] Pablo Fernández Alcantarilla | LNCS 7577 -KAZE Features[END_REF] algorithms. However, AKAZE is much more efficient in detection robustness, description and in the speed of calculation as it was created with high-performance algorithms in a pyramidal framework comparable to other algorithms of the same type. Even if AKAZE follows the same pyramidal steps as the other algorithms, the method used is very different. AKAZE integrates Fast Explicit Diffusion (FED) [START_REF] Grewenig | From box filtering to fast explicit diffusion[END_REF] systems for accelerated feature detection in a non-linear scale space. Moreover, a modified version of Local Difference Binary (M-LDB) [START_REF] Alcantarilla | Fast explicit diffusion for accelerated features in nonlinear scale spaces[END_REF] has been integrated into AKAZE. Unlike the old version, this modified version of LDB [START_REF] Yang | Local difference binary for ultrafast and distinctive feature description[END_REF] is a rotationally invariant, scaled descriptor and can exploit gradient information from the non-linear scale space.

The proposed registration method is shown in Figure 5. The first step is to extract the green channel (G) from the visible image, and the near infrared channel (NIR) from the infrared image (these channels were selected for their vegetation texture information). By using the normalization equation ( 1), the second phase normalizes the two spectral channels to improve their contrast.

I N ormalized = 255 × I-min(I) max(I)-min(I) (1)
where I N ormalized is a normalized source image I (visible or infrared), min(I) and max(I) are respectively the minimum and maximum grey level value of the image I.

The third step is to extract the points of interest and calculate their features from the two channels (by the AKAZE algorithm). Based on the features of the interest points, the fourth stage is to map each point of interest extracted in the G channel to the corresponding point in the NIR channel. In the fifth step, to eliminate some outliers, a first thresholding algorithm performs a preselection of inlier matching points, then a final selection is performed by the RANdom SAmple Consensus (RANSAC) algorithm [START_REF] Fischler | Paradigm for Model[END_REF]. To obtain the best setting for algorithms that contribute to removing the outliers, a dynamic setting (by threshold variation) method based on homographic matrix analysis is integrated into the registration system. This dynamic method (dynamic threshold) very significantly reduces the number of registration failures. Note that X is the coordinates of the source infrared image (x, y), X' is the coordinates of the registered infrared image (x', y') and H is the homographic matrix 2. In order to avoid the problems that lead to misregistration, a dynamic threshold regulation of the RANSAC algorithm was used, and the distance-based algorithm.

For each given value, a homographic matrix is estimated. The viability of this matrix is then tested by equation 3. This test is performed by projecting the end coordinates of the source image into the new space of the registered image.

If the result found is coherent (the distance between the old coordinates and the new coordinates does not exceed a certain threshold), this implies the end of the dynamic adjustment procedure, and the matrix tested is used to register the image. Otherwise, a new setting is made to repeat the same procedure. Once this step is finished, the pre-registered infrared image can be obtained by the homographic matrix validated by the dynamic adjustment procedure.

H =      1 + h 00 h 01 h 02 h 10 1 + h 11 h 12 h 20 h 21 1      (2)
X = HX (3)
where In order to reduce the registration error, an iterative method was implemented. After the pre-registration has been completed, an iterative phase starts from the result obtained, calculates the error by the Root Mean Squared Error (RMSE) [START_REF] Ross | Geopositional Statistical Methods Lockheed Martin Integrated Systems & Solutionsdoi[END_REF] between matched points which are calculated on the X coordinates with equation ( 4), and on the Y coordinates with equation ( 5), and then calculates their module by equation [START_REF] Chuche | Biology and ecology of the Flavescence dorée vector Scaphoideus titanus: A review[END_REF]. If the error decreases, another iteration is performed, if not, the iterative process stops and the best result, which is the result with the minimum RMSE, is kept.

285 RM SE x = N i=0 (x V ISi -x IRi ) 2 N (4) RM SE y = N i=0 (y V ISi -y IRi ) 2 N
(5)

RM SE = RM SE 2 x + RM SE 2 y ( 6
)
where (x V ISi , y V ISi ) and (x IRi , y IRi ) are respectively the "i" th coordinates of matching points between the visible and infrared images. N is the number of matches found between the visible and infrared images.

Dataset

This subsection presents the description of different classes, the labelling method, and the data generation using data augmentation.

Classes description

The dataset was composed of visible and infrared range images. In our case, the visible sensor was used to detect the presence or absence of chlorophyll in the crop, i.e. to detect any anomalies in the vegetation related to its discoloration during flowering and fruit set period. The infrared sensor used operates in the range of 850 nm. This wavelength was chosen for its sensitivity to changes in the different states of vegetation. The high reflectance generated by vegetation in the near infrared spectrum also makes it relevant for plant analysis. This class is generally characterized by yellow or brown color of the foliage in the visible range (in the case of an advanced symptom). This discoloration results from a problem with chlorophyll production. In the infrared range, the symptomatic class is characterized by a variation in reflectance between a diseased leaf and its neighbors. Visually it appears as specific texture, with which this class can be distinguished from the others. After merging the segmented visible and infrared images, two classes were created a "symptomatic intersection" class when symptoms were detected in the same location of visible and infrared areas (AND operator), and a "symptomatic union" class when two symptoms were detected either in the visible or in the infrared spectra (OR operator).

Semi-automatic data labelling

Due to the large amount of data and the difficulty of achieving accurate labelling, the data labelling process was performed by a semi-automatic method (see Figure 6). First, a sort of weak learning procedure was used to roughly detect the potential diseased areas using a sliding window over the image. Each To achieve the automatic labelling step, two LeNet5 datasets were created for visible and infrared images (from 480 samples of each range) using the translation and rotation data augmentation method (see Figure 7). Each dataset was organized into 4 classes (shadow, ground, healthy and symptomatic class), and contained 70, 560 patches of size 32 × 32 pixels (17, 640 samples for each class, among these samples, there were 14, 994 samples for training and 2, 646 samples for validation). Visible and infrared LeNet5 models were generated from these two datasets (see Figure 8).

Once LeNet5 models had been generated, the sliding window method was used to perform automatic labelling on UAV images with a size of 4608 × 3456 pixels. To obtain the best possible accuracy, the sliding window was set with a displacement step of 2 × 2. The automatic labelling operation required more than an hour and a half of runtime to label only a single UAV image.

Data augmentation

Due to the huge amount of data required to train a deep learning network, the lack of data (specially the symptomatic sample), and the difficulty of labelling images, several methods of data augmentation [START_REF] Dellana | Data augmentation in CNN-based periocular authentication[END_REF] were used (see Figure 10). For that purpose, a sub-sampling between 0.5 to 1 of the real scale, and an oversampling between 1 to 1.5 was performed with a step of 0.25. The fourth technique of data augmentation consisted in modifying the brightness, to make the network less sensitive to light changes. As the image acquisition was done outdoors, the brightness parameters were uncontrollable because of changes in the weather. Thus, coefficients between 0.8 to 1.2 with a step of 0.1 were multiplied by the grey levels of the images to generate dark, normal, and bright effects.

Training and testing data

By using data augmentation methods, a dataset of 105, 515 and 98, 895 patches (360 × 480 pixels for each patch) was generated for the visible and infrared range respectively. This dataset was used to train and validate the SegNet models. Among the dataset patches, 85% (89, 688 for the visible range, and 84, 061 for the infrared range) was randomly selected for training and the remaining ones (15%) were used for validation. To evaluate and test the SegNet models, 200 patches for each range (without undergoing data augmentation), taken from another area at two different periods of time were used to test the visible and infrared models, and the fusion between them.

Segmentation and fusion

This subsection presents the visible and infrared image segmentation system, the deep learning architecture used in this process, the labelling and training method, and finally, the overall operation.

Deep learning segmentation

The CNN architecture has been very successful in the pattern recognition and computer vision fields. Since then, there has been a continuous evolution of CNN architectures. The new architectures have become deeper, but also new types of architectures have emerged that directly segment an image, such as the SegNet [START_REF] Badrinarayanan | SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation[END_REF] architecture, used in the present study. To segment an image, the SegNet architecture (Figure 11) operates through two opposite phases, an information encoding phase and a decoding and classification phase (Table 2). The encoding phase is in fact a classic CNN architecture, usually with a VGG-16 [START_REF] Zhang | An empirical-theoretical calculation method and fullscale verification test for the ultimate bearing capacity of an expanded pedestal uplift pile[END_REF] architecture. The coding phase consists of three types of processing, namely con- At this level, a pixel-wise segmentation is performed by the Softmax function.

In the present study, generation of the disease map of a vineyard field was considered as a four-class segmentation problem in the visible and infrared range.

The objective was to build a SegNet model capable of differentiating between shadow, ground, healthy and symptomatic (visible and infrared) classes in the two spectral bands. The distinction between each class is mainly based on variations in color, texture, spectral information and spatial relative position of each class. This important information must be extracted by the SegNet network during the encoding phase, also called the feature extraction phase, then rebuilt and segmented by the decoding phase.

Fusion of multimodal image segmentation

The fusion of segmentations was performed in order to obtain a disease map with more robust results. To generate a disease map, each pixel of the image segmented in the visible range was compared with the pixel of the same position in the infrared range. Here, three main cases are considered. The first one is that the two pixels represent the symptomatic class. In this case, the result is a symptom intersection class. The second case is when the pixel is classified as symptomatic in the infrared range, and healthy in the visible range. In this case, the resulting class is symptomatic infrared (This may be a case where the disease has not yet affected the visible range by discoloration of the leaves). The third case is when the pixel is classified as healthy in the infrared range, and symptomatic in the visible range. The resulting class is visible symptomatic.

To evaluate the disease map, two cases were assessed. The first one is the case described in the previous paragraph. It is named fusion by intersection and symbolized "Fusion AND"; the AND operator means that the symptom is considered to be detected if it is present in both visible and infrared images.

The second case is named fusion by union and is symbolized "Fusion OR". As the name indicates, this case unites visible and infrared range detections with the OR operator.

Experimentation

This section details the experiments, testing, validation and interpretation of the results. The algorithms were developed under the Python 2. The experimental section is divided into two subsections. The first one concerns the visible and infrared image registration, and the second one details the segmentation and fusion steps.

Evaluations of image registration algorithm

Performance measurement was carried out by computing the RMSE Eq. 4, 5 and 6 between the points of interest matched, in pixel units. RMSE provides

After seven iterations

Standard registration Optimized registration batch was composed of 5 patches and with a number of iterations of 100, 000.

Lastly, two SegNet models were generated, the first one for the visible and the second one for the infrared. These models were evaluated by quantitative and qualitative methods.

The quantitative evaluation was carried out on two different times from the second area of P1 (not used for training). The results were compared to the ground truth, to evaluate the method.

In order to show the segmentation quality, a qualitative evaluation was carried out on some images from the second area of P1 (Figure 14), and some images from P2 (Figure 15). To realize this step, the UAV image was cut into patches of size 360 × 480 pixels. These patches were segmented by the Seg-Net network, then stitched together to find the original size of the UAV image (4608 × 3456 pixels).

Performance measurement

The segmentation performance was measured by two methods. The first one (presented in Table .4) is based on the leaf-level (pixel-wise) computation of the Recall (eq.7), precision (eq.8), F1-Score (eq.9)/Dice coefficient (eq.10) and Accuracy (eq.11) for each class (shadow, ground, healthy and symptomatic).

The second evaluation is based on the grapevine-level (presented in Table.5).

The pixel-wise segmentation measurement does not provide information about detection at the grapevine level; it merely indicates whether a given grapevine is infected or not. This measurement uses a sliding window with a size of 64×64 pixels (corresponding to the average size of a grapevine in the studied plots).

Inside this window, only the dominant class in the ground truth is evaluated.

If there is a match between the ground truth and the SegNet estimation, it is considered as true positive, otherwise it is false positive. At the end of this process, the same measurement is computed.

Recall = T P T P +F N (7)
P recision = T P T P +F P where TP, TN, FP and FN are the number of samples for "True Positive", "True Negative", "False Positive" and "False Negative". For the Dice equation, X is the set of ground truth pixels and Y is the set of pixels classified by the SegNet classifier.

Discussion

The first question of this study was to determine the ability of multispectral drone imaging to map grapevine Mildew symptoms using machine learning Table 4: The leaf-level (pixel-wise) average result on two temporal tests by measuring Recall (Rec.), Precision (Pre.), F1-Score/Dice (F1/D.) and Accuracy (Acc.) on the performance of visible, infrared image segmentation and fusion (values presented in percent). Note that: "Fusion AND" and "Fusion OR" are the cases where their symptomatic classes are respectively the intersection and the union of the symptomatic visible and infrared classes. Table 5:

The grapevine-level average result on two temporal tests by measuring Recall (Rec.), Precision (Pre.), F1-Score/Dice (F1/D.) and Accuracy (Acc.) on the performance of visible, infrared image segmentation and fusion (values presented in percent). Note that: "Fusion AND" and "Fusion OR" are the cases where their symptomatic classes are respectively the intersection and the union of the symptomatic visible and infrared classes. approaches. This led to studying imaging modalities in the visible and infrared spectral domains, since several studies have shown the interest of these domains for symptom detection. As our system consisted of two cameras for each modality, image alignment was required. We were, therefore, led to develop an algorithm for image registration and then to use deep learning segmentation to detect the affected surfaces in the vineyard. Another question of this research was how, through the deep learning approach, we can combine the two types of images to delineate symptomatic areas as precisely as possible. Thus, the rate of affected areas can be obtained at the leaf scale or at the vine plant scale. The following sections first discuss the results of the registration and then those of the image segmentation.

Image registration

Figure 12 shows the qualitative result of the registration. On the left and in the middle are the visible and infrared images respectively. The images were taken at a very short time interval (less than one second) and in the same field of view. After registration, two black areas can be observed on the left and at the bottom of the image of the right. The two areas were captured only by the visible sensor, which explains why there are no equivalents in the infrared image registered at the moment of superposition. There are also areas captured only by the infrared sensor, but due to the fact that the infrared image is registered to the visible image, these areas are not displayed on the result.

An example of how the proposed registration method operates is shown in Figure 13. In the image on the left (image registered with the standard method) there is a certain shift between the vinerows present in the visible and infrared images. This is due to a lack of matched points between the two images. However, after each iteration, some new correspondence points are detected thanks to the dynamic threshold, and gradually, seven iterations later, a good alignment of the vinerows is achieved as shown in the image on the right.

The quantitative results obtained during the registration experiments are presented in Table 3. This table reports a statistical study on a dataset of 150 images, for a comparison between the standard registration method used in [START_REF] Sreenu | Control of Brushless DC Motor with Direct Torque and Indirect Flux using SVPWM Technique[END_REF][START_REF] Javadi | An Automatic Robust Image Registration Algorithm for Aerial Mapping[END_REF] and the proposed optimized registration method. The values obtained are expressed in "pixels" for the RMSE error measurement, in "seconds" for the runtime and in "times" for the iteration. For each line, the figures in bold show the best results obtained. For standard registration, an average error of 3.29 pixels was obtained (this result corresponds to the range of results obtained in [START_REF] Onyango | Accurate estimation of orientation parameters of UAV images through image registration with aerial oblique imagery[END_REF] by the same algorithm), compared to 2.43 pixels for the optimized method, i.e.

an average error reduction of 0.86 pixels over the entire dataset. This error reduction can be explained by the appearance of new correspondence points, in subsequent iterations, between the two images. These points are used to calculate a new homographic matrix, which corrects the registration slightly, and reduces the RMSE error. However, there are special cases where the optimized method does not reduce the error, in which the result of the optimized method is cancelled and the output of the standard registration is kept. The best score obtained by the standard method yields an error of 1.13 pixels, whereas with the optimized method the minimum error is 0.87 pixels. However, both methods can produce larger errors, as we can find errors up to 9.75 pixels with the standard method and 9.02 pixels with the optimized method. Note that our disease detection system performs better with an RMSE value ≤ 5; RMSE values between 5 and 10 may in some cases reduce the accuracy of disease localization.

The runtime measurement is shown in Table 3. For the standard registration, an average runtime of 92 seconds was obtained, versus 139 seconds with the optimized method. This increase in runtime is due to the additional processing performed by the optimized method (3.12 iterations on average). This is not the only reason, however: with the standard method, there is a very significant difference (80 seconds) between the minimum and the maximum values. The difference is due to the number of interest points detected in an image by the AKAZE algorithm: the larger the number of interest points detected, the longer the processing time. In other words, there is a direct relation between these two parameters.

The proposed algorithm based on the AKAZE detector was tested by other feature extraction algorithms such as SIFT, SURF, ORB and KAZE. The results obtained by SIFT showed a slightly higher error than those obtained by AKAZE, and the runtime was between 2 to 10 times longer than AKAZE, confirming the study by Tareen et al. [START_REF] Tareen | A comparative analysis of SIFT[END_REF]. For the other algorithms, several cases of failure were identified. They are mainly due to the lack of correspondence between the two images, which implies a sensitivity of these algorithms to the differences of modality between the two images. Another algorithm developed by Yang et al. [START_REF] Yang | Multi-temporal remote sensing image registration using deep convolutional features[END_REF], which is based on feature extraction through a deep learning architecture was tested, and several problems were observed. Not only did the deep learning method use a large amount of RAM memory, making it very difficult to register high resolution images, but even with a decrease in the resolution of the images, the algorithm cannot find good matches, which leads to the failure of the registration process. This is due to the difference in image modalities.

Some other Area-based registration algorithms were also tested, such as:

Normalized Cross-Correlation (NCC) [START_REF] Wang | Efficient registration of optical and IR images for automatic plant water stress assessment[END_REF] and Phase-Correlation (PC) [START_REF] Erives | Automated registration of hyperspectral images for precision agriculture[END_REF]. But due mainly to the presence of some deformations on the image dataset used, these algorithms did not manage to correctly register most of the images. Another disadvantage of these algorithms is that they are very time consuming, and were therefore discarded.

Image segmentation and fusion

The qualitative results given in figures 14 and 15 represent the results on the vineyards of P1 and P2 respectively. For the P1 plot (Figure 14), a large part of the untreated plot was contaminated by Mildew. This presence of disease is well detected by segmentation. As can be observed, in most cases, the symptoms are better detected in the visible range (coded in yellow) than in the infrared range (coded in orange). This hypothesis was confirmed by the quantitative results in Table 4. Of course, there are in some cases false detections in both areas. However, when symptoms are detected in both ranges it is likely to be true detection.

For the P2 plot (Figure 15), the vineyard is healthy, and the SegNet estimation generally matches the ground truth. However, in this example of 3 images, some misclassification of symptomatic areas in the visible range can be observed. This generally occurs when there is a gap in the vegetation (a plant missing in a row, for example) and the color of the soil is similar to that of a symptomatic plant (brown or golden). Also, it is usually explained by the edge effects of the sliding window, because the information is not complete on some borders (these effects are not always present in the outputs). In other cases, the symptom class may be assigned when yellow is mixed with green, as this case is usually comparable to that of symptomatic leaves. To counter this problem, it is necessary to check the neighboring images that cover this area.

If the symptom is detected in all or most of the images, there may be a real symptom, otherwise, it is a false detection. Misclassification of the other classes can also occur. Indeed, in these examples, some misclassification of the grass can be observed which is detected as healthy vine or symptom class (in the 1st example of Figure 15). Some confusion between the ground and the shadow class can also occur because of the low ground brightness (in the 2nd and 3rd examples of Figure 15).

Figures 16 and17 show an example of SegNet segmentation and the fusion compared with the ground truth. In Figure 16 the area is healthy, so there are no samples of the symptom class. In this case, it can be observed that the fusion is identical to the visible and infrared estimation (idem for the ground truth).

In Figure 17, in contrast, which is an area almost completely contaminated by Mildew, it can be seen that the ground truth is identical in both spectra, apart from the color code which is different. But for the SegNet output, the result for the symptomatic class detection is not identical in the two ranges, and consequently the symptomatic visible and infrared classes are merged. In addition, fusion by intersection is generated by the AND operator between the two segmentation outputs.

The quantitative results obtained for the visible, infrared, fusion by intersection and union segmentation experiments are presented in Table 4 and5, respectively for leaf-level and grapevine-level. They show the results obtained in terms of the Recall, Precision, F1-Score/Dice coefficient and Accuracy measures, expressed in percentages.

As shown in the "Accuracy" column of In addition to measuring the affected areas at leaf level, the second type of assessment consists in testing the detection at vine plant level, as this helps to better manage certain operations in the vineyard. The results presented in Table 5 give better insight into the detection of symptoms at the grapevine-level.

The results show that the detection of symptoms in fusion by union is much better (detection exceeds 92.81%) compared to the fine scale detection (leaf level), followed by the visible range (91.50%), then the infrared range (81.66%).

As can be seen, the results of infrared and fusion by intersection are less than the pixel-wise evaluation (Table 4). On the other hand, an increase in precision for the cases of fusion by union and the visible range can be observed.

Results obtained by the proposed method are likely to be consistent with several studies in the field of remote sensing [START_REF] Du | Training SegNet for Cropland Classification of High Resolution Remote Sensing Images[END_REF][START_REF] Jiang | RWSNet: a semantic segmentation network based on SegNet combined with random walk for remote sensing[END_REF][START_REF] Marmanis | Classification with an edge: Improving semantic image segmentation with boundary detection[END_REF][START_REF] Chai | Cloud and cloud shadow detection in Landsat imagery based on deep convolutional neural networks[END_REF][START_REF] Audebert | Segment-before-detect: Vehicle detection and classification through semantic segmentation of aerial images[END_REF][START_REF] Milioto | Real-Time Semantic Segmentation of Crop and Weed for Precision Agriculture Robots Leveraging Background Knowledge in CNNs[END_REF] using the SegNet network. Indeed, the overall accuracy range obtained by these studies is between 70% and 90%. In addition, it has been observed that when the surface area is large, the detection result is better. Conversely, the smaller the surface area, the more difficulty the SegNet network has in correctly detecting the diseased area. A possible explanation for this might be the loss of the resolution information of small areas during the downsampling and upsampling operations in the SegNet network. Another reason why the results are limited is the difficulty of realizing an accurate ground truth for training and testing the network, but also the difficulty identifying an area in a low resolution (1cm/pixel) (in the case of images taken by a UAV at high altitude). Other remote sensing applications with large database studies that have tested and compared several types of deep learning architecture [START_REF] Ma | Fully convolutional network for rice seedling and weed image segmentation at the seedling stage in paddy fields[END_REF][START_REF] Chai | Cloud and cloud shadow detection in Landsat imagery based on deep convolutional neural networks[END_REF] such as FCN, U-Net, DeepLab, PSPNet, etc., obtained the best semantic segmentation results with the SegNet network.

In the proposed system, the fusion by intersection of the two modalities indicates the locations where symptoms were located at the same position in the visible and infrared images. In other words, fusion provides important information about areas where the system confidence is higher for disease detection.

Moreover, it also provides information about areas where the disease has been detected only in one range (visible or infrared). Therefore, even after the establishment of the method for disease detection, fusion by intersection remains a more reliable class than the symptomatic classes detected in the visible or infrared range.

RGB and multispectral images were combined as one input then fed to the SegNet. The results obtained were poorer. This is due to the registration of the visible and infrared images. In fact, there is always a small random shift between the multispectral images, which implies that the same pixels are never aligned exactly in the same position.

Runtime system

Table 6 presents the runtime results of all stages of the disease detection system. For the image registration step, the average runtime value was used to evaluate the overall system, because the image registration runtime is variable from one pair of images to another. For the SegNet segmentation step, the runtime was multiplied by two (× 2) because the process must segment both images (visible and infrared) and the GPU can only handle one process at a time, unlike registration and fusion operations, where the processing is joint for the two images (visible and infrared). Unlike image registration, the runtime of the SegNet on a UAV image is constant, at 140 seconds. This value is the same for both visible and infrared images. The fusion between the two segmented images takes less than 2 seconds; this runtime value includes the computation and saving the fusion file. The runtime of the disease detection system varies according to the image registration method chosen. This implies that to improve the accuracy of the results, an additional average processing of 47 seconds per image is necessary.

Conclusion

In this study, a new method based on optimized image registration and a deep learning segmentation method has been proposed for detecting vine disease using multimodal UAV images (visible and infrared ranges). The method

Figure 1 :

 1 Figure 1: The study vineyard seen by satellite. Plot (P1) in picture (a) and plot (P2) in picture (b).

Figure 2 :

 2 Figure 2: The plot (P1) seen from the ground.

Figure 3 :

 3 Figure 3: The acquisition equipment used in this study. The quadcopter UAV drone (a) and the high-resolution Survey2 sensor (b).

Figure 4 :

 4 Figure 4: Overview of disease detection system in grapevine fields.

( 1 +

 1 h 00 ) and (1 + h 11 ) are stable scale factors respectively in the X and Y direction only. h 01 and h 10 are scale factors respectively in the X direction relative to the Y distance from the origin and Y direction relative to the X distance from the origin. h 02 and h 12 are respectively translation in the X and Y direction.h 20 and h 21 are relative scale factors X and Y respectively as a function of X and Y.

  with last infrared image preregistered

Figure 5 :

 5 Figure 5: Proposed method for non-rigid registration of multimodal (visible and infrared) image.

  The visible and infrared datasets were labelled based on four dominant classes in the visible and infrared ranges, fusion by intersection, and fusion by union. The first class represents the shaded areas in the vine and on the ground (all the dark areas). This class does not reflect light, which means that no relevant information can be drawn from it. The second class represents the ground; it can be an area of weeds, grass or any type of soil. The third class represents the healthy vegetation of the vine. This class has a green color in the visible range and has a high reflectance level compared to the previous classes in the infrared range. Finally, the fourth class represents the symptomatic vine.

  block was then classified by LeNet5 network for pre-labelling. Then, manual labelling was performed on the basis of the ground truth provided by technicians in the field. Segmentation steps by LeNet5 Model: 1. Moving the sliding window. 2. Patch window estimating by using LeNet5. 3. Writing the estimated label on the center of the sliding window. 4. Moving the pointer and return to (1).Pre-labelling process by segmentation using LeNet5 ModelManual labelisation Overview of semi-automatic ground truth for visible and infrared images

Figure 6 :

 6 Figure 6: Overview of semi-automatic pixel-wise labelling for visible and infrared images.

Finally, manual correctionFigure 9 :

 9 Figure 9: Example of semi-automatic labelling. The pre-labeling was carried out by the LeNet5 model using the sliding window method for segmentation. Manual correction was performed using Paint.Net software.

First, two registeredFigure 10 :

 10 Figure 10: Data augmentation process of the visible and infrared datasets for training the SegNet architecture.

  volution layers, ReLU layers (ReLU is an activation function for removing the negative values that result from convolution and deconvolution. It is commonly used in deep learning networks, because it performs better than other activation functions) and MaxPooling layers (non linear sub-sampling function). Decoding consists of the same types of processing except for the convolution layers which are replaced by deconvolution layers, and the MaxPooling layers which are replaced by Upsampling layers. It is in the decoding part of the network that the segmented image is formed, until the final decoding layer is reached.

  7 development environment using the Tensorflow 1.8.0, NumPy 1.16.2 and OpenCV 3.0.0 libraries. To run and evaluate the runtime of our algorithms, we used the following hardware; an Intel Xeon (R) W-2123 (a) 3.60 GHz×8 processor (CPU) with 32 GB of RAM, and a graphics processing unit (GPU) NVidia GTX 1080Ti with an internal RAM of 11 GB under the Linux operating system Ubuntu 16.04 LT S (64 bits).

Figure 13 :

 13 Figure 13: Correction of the shift by seven iterations.

F 1 - 9 )

 19 Score = 2 Recall×P recision Recall+P recision = 2T P F P +2T P +F N (Dice = 2|X∩Y | |X|+|Y | = 2(T P ) (F P +T P )+(T P +F N ) = 2T P F P +2T P +F N (10) Accuracy = T P +T N T P +T N +F P +F N (11)

Figure 14 :

 14 Figure 14: Qualitative segmentation results by the SegNet method for the P1 vineyard. On the left, images from UAV on the right, the segmentation result of these images. The color code of the segmentation is: Black: Shadow, Brown: Ground, Green: Healthy, Yellow: Visible symptom, Orange: Infrared symptom, Red: Symptom intersection.

Figure 15 :

 15 Figure 15: Qualitative segmentation results by the SegNet method for the P2 vineyard. On the left, images from UAV on the right, the segmentation result of these images. The color code of the segmentation is: Black: Shadow, Brown: Ground, Green: Healthy, Yellow: Visible symptom, Orange: Infrared symptom, Red: Symptom intersection.

Figure 16 :Figure 17 :

 1617 Figure 16: Example of segmentation and fusion of a healthy area. (a): Visible image, (b): Infrared image, (c): Visible ground truth, (d): Infrared ground truth, (e): Fusion ground truth, (f): Visible SegNet estimation, (g): Infrared SegNet estimation, (h): Fusion of segmentation results.

  comprises three steps. The first one is the image alignment, where an iterative algorithm based on an interest points detector has been developed. The second step is the segmentation of visible and infrared images based on the SegNet architecture to identify four classes: shadow, ground, healthy and symptomatic vine. Lastly, the third step consists in generating a disease map by fusion of the segmentations obtained from the visible and infrared images. This study showed that the proposed method enables the detection of vine symptoms using information from images of visible and infrared spectra. It provides a framework for the exploration of earlier detection and mapping of vine diseases. One of the limitations of this research is the small size of the training sample which reduced the performance of the deep learning segmentation. In future work, the following improvements could be made. The segmentation method can be improved by enriching the dataset (diversification of disease samples), and also by testing other deep learning architectures for segmentation. Another possibility is the use of 3D information from the vine canopy, thus reducing false detection and improving the accuracy of image registration.

Table 1 :

 1 Description of the study vineyard.

	Types of information	Description (P1)	Description (P2)
	Surface	1.8 hectares	1.5 hectares
	Altitude	110 meter	114 meter
	Planting year	1976	1991
	Variety	Malbec	Sauvignon
	Vine plant spacing	1 meter	1 meter
	Interline distance	1.5 meter	1.5 meter
	Slope	7%	0%
	Exposure	North-South	Northwest-Southeast
	Coarse fraction	25%	17%
	Soil Organic Carbon	4.46%	3.94%
	Soil type	Silty-Sand	Sandy-Loam

Table 6 :

 6 Results on runtime performance (expressed in seconds) for the entire system as a function of the average runtime of standard and optimized registration.

	Class name	Shadow		Ground		Healthy		Symptomatic	Total
	Measure	Rec.	Pre.	F1/D. Rec.	Pre.	F1/D. Rec.	Pre.	F1/D. Rec.	Pre.	F1/D. Acc.
	Visible	94.00	93.42	93.63	97.39	97.94	97.66	95.16	85.20	89.91	90.15	92.97	91.50	94.41
	Infrared	97.53	79.97	87.55	97.30	95.41	96.32	93.72	69.19	79.19	70.49	96.92	81.66	89.16
	Fusion AND	94.01	86.62	89.96	97.41	97.89	97.65	93.81	87.55	90.56	66.92	73.12	68.03	88.14
	Fusion OR	94.00	94.03	93.95	97.39	97.94	97.66	93.81	89.65	91.68	92.91	92.78	92.81	95.02
		Step			Registration	SegNet seg.		Fusion		Total
	Standard registration	92		140 × 2		2		374
	Optimized registration	139		140 × 2		2		421

Table 4 ,

 4 the different classes were generally better detected from the visible image (accuracy of 85.13%) than from the infrared image (accuracy of 78.72%). This difference in result is due to the fact that the visible image provides a better colorimetric description than the infrared image for the different classes studied. The fusion by union gave a result of 90.23%. This result is better than the visible image, because the method takes the best of visible and infrared information. It can be noticed that symptoms appear at different locations in the visible and infrared spectral ranges. One interesting finding is that detection in the two ranges is complementary, since fusion by union increases the detection performance. The fusion by intersection yielded a score of 82.20%. This result is less than that of the visible range, because the method is conditioned by the intersection of the visible and infrared, and in this case, it is the infrared result that decreases the result. The fusion

by intersection provides important information about the position of the joint symptom detection in the visible and infrared images. This finding can be used to strengthen the robustness of Mildew detection, where detection can be considered reliable if it is observed in both types of images. Besides, the fusion by union gives an idea about the quantitative detection.

lighting (acquisition times were between 11:30 and 13:30 to avoid the shadow

P1. During this phase, the network uses patches of size 360 × 480 pixels. Each

Acknowledgment

This work is part of the VINODRONE project supported by the Region Centre-Val de Loire (France). We gratefully acknowledge Region Centre-Val de Loire for its support.

Shadow

eased ones. By referring to the ground truth, each vine was referenced with its coordinates in the field. Then, this information was used to label the areal images. Figure 9 shows an example of the semi-automatic labelling steps. Training of the SegNet model was carried out using data from the first area of