Yliess Hati
email: yliess.hati@devinci.fr

Francis Rousseaux
email: francis.rousseaux@univ-reims.fr

Clément Duhart
email: duhart@mit.edu

Clement Duhart

TEXT-DRIVEN MOUTH ANIMATION FOR HUMAN COMPUTER INTERACTION WITH PERSONAL ASSISTANT

come

INTRODUCTION

Recent developments in the Artificial Intelligence (AI) communitymore precisely in Deep Learning -re-enhanced affective computing with the promise of new communication layers between human and machine. This research area explores how computers can sense, analyze, generate, and express affect features as humans do. Sense and analysis of users' emotional states received much attention in the research community, especially for facial expression recognition [START_REF] Li | Deep facial expression recognition: A survey[END_REF], body gesture recognition [START_REF] Noroozi | Survey on emotional body gesture recognition[END_REF], speech recognition, and natural language processing [START_REF] Weninger | Emotion Recognition in Naturalistic Speech and Language -A Survey[END_REF]. Usually, the identification of complex human affect expressions requires the use of multimodal frameworks or data fusion techniques [START_REF] Poria | A review of affective computing: From unimodal analysis to multimodal fusion[END_REF], and the current state of the art allows the software to be responsive to the user's emotional states. However, affective Human Computer Interface (HCI) would also benefit from giving this kind of abilities to the computer. On the one hand, the computer should be able to generate an internal affective state in response to the user's interactions. On the other hand, it should be able to express it more naturally and realistically. Therefore, the user's biases regarding computer interaction could be reduced and would allow more credible communication loopback between human and machine. Several contributions have studied the use of emotions during interactions with virtual agents, especially in negotiation [START_REF] De Melo | The effect of expression of anger and happiness in computer agents on negotiations with humans[END_REF][START_REF]The effect of virtual agents' emotion displays and appraisals on people's decision making in negotiation[END_REF]. At our best knowledge, all of them used hard-coded fixed sets of expressions, whereas human's emotions is a continuous space. Our work aims to provide visual and acoustic expression abilities to computers. Figure 1 presents our Figure 1: Personal Assistant based on our text-driven mouth animation system in a workspace setup.

workspace setup where users can interact with a personal assistant using our text-driven mouth framework. In this contribution, we propose an end-to-end architecture for voice synthesis with its associated mouth animations. This work is a preliminary step toward affect controls.

RELATED WORK

Over the last decade, facial animation received considerable attention from industries and research communities for reducing their production cost and their fidelity. At the early days of this filed, facial animations were made by hand and required the expertise of professional animators. The 3D models were first rigged, weighted and then animated frame by frame. This process is time-consuming and varies with the animators. More recently, the use of performance capture allowed to semi-automatize facial animations. Facial movements are recorded with specific software and hardware by tracking markers on human actors in real-time. The collected data is then transferred to the 3D character model. The results are highly dependent on the actor morphology and appearance and require intensive cleaning [START_REF] Ma | Semantically-aware blendshape rigs from facial performance measurements[END_REF][START_REF] Smith | Emotion challenge: building a new photoreal facial performance pipeline for games[END_REF][START_REF] Weise | Realtime performance-based facial animation[END_REF]. Nowadays, thanks to the recent enhancements in Deep Learning, video and audio based approaches are gaining popularity.

The overall process requires the system to produce mouth ani- mations synced with the speech. One method is to define a dictionary of visual mouth forms called visemes. Then, the system needs to learn how to associate those visemes with the right phonemes and smooth the result using interpolation. Such approach discretizes the visual and acoustic spaces to bind them. Early works proposed the use of Hidden Markov Model [START_REF] Cosker | Speech driven facial animation using a hidden markov coarticulation model[END_REF][START_REF] Xie | Speech animation using coupled hidden markov models[END_REF] whereas modern ones are using Deep Learning techniques [START_REF] Zhou | Visemenet: Audio-driven animator-centric speech animation[END_REF][START_REF] Taylor | A deep learning approach for generalized speech animation[END_REF][START_REF] Cao | Expressive speech-driven facial animation[END_REF][START_REF] Edwards | Jali: An animator-centric viseme model for expressive lip synchronization[END_REF]. One limitation is the lack of expressiveness in the mouth articulations, limiting the realism of such generated animations. Moreover, audio and phonemes alignment is a complicated task and is not always feasible. Nowadays, the Deep Learning community has been able to solve similar issues in translation. Instead of discretizing the input and output space to find bindings, the system learns how to map those spaces together directly. Hence, results are consistent in the output space. In the animation field, recent contributions present significant improvements using such approaches known as audio-driven or speech-driven facial animation. Such frameworks allow to synthesize facial movements and can include an emotional dimension [START_REF] Pham | End-to-end learning for 3d facial animation from raw waveforms of speech[END_REF][START_REF] Karras | Audio-driven facial animation by joint end-to-end learning of pose and emotion[END_REF]. Unfortunately, the authors did not share their dataset, limiting reproducibility for future contributions.

Recently, ObamaNet from Rithesh and al. introduced the first approach for text-driven mouth animations [START_REF] Kumar | Obamanet: Photo-realistic lip-sync from text[END_REF]. Using text as input, their model can generate an audio waveform and its photorealistic lip-sync frames. Their results are impressive and can lead to future approaches. However, this method is not suitable for other applications requiring the control of a 3D character model.

Our contribution includes:

• A methodology for the creation of text-driven mouth animation datasets from a video bank, including audio. First, the speaker's face is automatically detected and cropped from the video. Then, facial 3D landmarks are identified and projected into an invariant space. Finally, the mouth's key points are extracted and normalized, providing natural synchronicity with the audio. • A new Deep Learning pipeline for text-driven mouth animation following best practices from state of the art. The speech synthesis module has been replaced by more recent contributions. We also introduce a new module to focus on 3D landmarks controls instead of 2D rendered frames.

• An experiment evaluating how our proposed generated mouth visuals can impact users' comprehension during a listening test and their realism during a blind test compared to landmarks extracted from real videos.

For research reproducibility, we will soon publish our generated dataset as well as our source code with public access.

ADMA DATASET

In this section, we describe the methodology used to generate our dataset ADMA-TED for Audio-Driven Mouth Animation (ADMA) TED based on the Lip Reading Sentence (LRS3-TED) dataset [START_REF] Afouras | LRS3-TED: a large-scale dataset for visual speech recognition[END_REF]. This dataset is composed of 400 hours of TED and TEDx English talks videos distributed over 4004 videos for training and validation, and 451 for testing. Each video varies from 1 to 6 seconds and is cropped on the speaker's face with a 224 by 224 pixels resolution. This dataset has been chosen for its diversity and quality, such as the faces are almost always visible continuously. The rejection rate is lower than 1% and concerns recordings with microphone glitches or where the face is not visible enough. For each video, we provide 20 normalized 3D mouth landmarks every 40 ms to ensure continuous face tracking during the sentence pronunciation, as illustrated in Figure 2.

Mouth Landmarks

Face Alignment Network (FAN), proposed in Bulat and al. [START_REF] Bulat | Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans[END_REF], is a popular model for face landmark inference on pictures. It has been applied on the entire LRS3-TED dataset at 25 FPS rate, providing 68 3D face landmarks for each frame as illustrated Figure 3.

Only mouth landmarks are extracted, as illustrated in Figure 4. They are then projected into a head rotation invariant and normalized space. We defined the top of the nose and the chin as our y axis, both eye's landmarks as our x axis and their cross product as our z axis. These axes define the face referential. Using the transformation matrix T M in Eq. 1, the mouth's landmarks are projected into a new front-facing referential. These landmarks are normalized between [0; 1] with the center of the mouth located at 0.5 in each axis. x 0 y 0 z 0 0

x 1 y 1 z 1 0 x 2 y 2 z 2 0 0 0 0 1 3 7 5 (1)
Finally, as people possess different types of mouth, identity is removed by computing the median lips thickness over the entire dataset to reduce this bias. Depending on given thresholds, some videos are rejected for being out of the maximum and minimum range of the mouths' opening and closing.

Audio Processing

Each video's audio channel from LRS3-TED is sampled at 16kHz. Their Mel-spectrogram are computed using 32 frequency bands, 128 hop length, and 512 window size. Each extracted mouth is associated with a centered 64 window size of the Mel-spectrogram and filtered to remove high-frequency glitches. Each datum of the dataset contains a spectrogram with its associated mouth landmarks. The dataset is composed of 1,032,219 elements for training and 35,473 for testing.

END-TO-END NEURAL NETWORK ARCHITECTURE

Our end-to-end neural network for text-driven mouth animations is based on the architecture proposed by Rithesh and al. [START_REF] Kumar | Obamanet: Photo-realistic lip-sync from text[END_REF]. As state of the art has improved over the years, we have upgraded the Text-to-Speech (TTS) module. Our neural module for audio-driven mouth 3D landmarks regression is connected to the end of the processing pipeline, as illustrated in Figure 6. The 20 3D mouth landmarks are used in a 3D engine to compute mouth's vertices and normals frame by frame using 3D spline interpolation, as shown in Figure 5. The final rendering is achieved with the use of a Phong shader.

Mouth Landmark Regression

This section details the mouth landmarks regression module in charge of estimating mouth 3D landmarks coordinates conditioned on 40 ms Mel-spectrograms from speeches.

For this task, we considered two aspects: the mouth landmarks 3D spatial positions and their dynamics over time. Hence, our From left to right and top to bottom each 3D plot respectively correspond to interpolation factors of (8, 1), [START_REF] Pham | End-to-end learning for 3d facial animation from raw waveforms of speech[END_REF][START_REF] Poria | A review of affective computing: From unimodal analysis to multimodal fusion[END_REF], (32, 8) and (64, [START_REF] Pham | End-to-end learning for 3d facial animation from raw waveforms of speech[END_REF] with horizontal interpolation first and vertical interpolation second. model contains three stages. The first stage learns frequency correlations in a compact feature maps representation over time. Then, the second stage determines the correlation dynamic between these step representations. Both stages use convolution layers with rectangle kernels such as time is considered as a spacial dimension instead of a sequence, allowing faster training and inference performances. Stride is preferred over pooling layers to keep as much information as possible. Finally, these compact features are used to train a Mean Squared Error (MSE) regression to estimate the 3D normalized landmarks positions.

Text-Driven Mouth Skeleton

Rithesh and al. [START_REF] Kumar | Obamanet: Photo-realistic lip-sync from text[END_REF] proposed the first contribution of end-to-end neural architecture for text-driven lip-syncing. Their work inspired our proposed architecture. We upgraded their architecture with state of the art contributions and extended it from 2D space coordinates to 3D ones. We replaced the Char2Wav module by Tacotron2 [START_REF] Shen | Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions[END_REF] to convert text input into a Mel-spectrogram and WaveGlow [START_REF] Prenger | WaveGlow: A Flowbased Generative Network for Speech Synthesis[END_REF] for phase reconstruction.

Neural Architecture Pipeline

Our proposal is an end-to-end neural architecture for text-driven mouth animations composed of different modules. Each one of those is a crucial step because of cumulative approximation errors throughout the entire model. During development, we tried different approaches for the pipeline. One of them was to connect our CNN regressor directly to the Tacotron2 output. However, the output resolution was not sufficient to allow our model to learn the mouth landmarks properly. The Mel-spectrogram use by our architecture is computed using an STFT on the waveform produced by WaveGlow, as illustrated in Figure 6, to allow control on its resolution. First one is in charge of learning frequency correlations at each time step whereas the second stage learns their dynamics according to the incoming Mel-spectrogram. Finally, the last stage learns a regression between these frequency dynamic feature maps and the mouth landmark positions.

Layer

Training

Our model has been trained for 5 hours over 1000 epochs using a single Nvidia 980M 8Go GPU. Using the Adma optimizer with a 1e 3 learning rate and (0.9, 0.999) betas, we achieved an error of 1e 3 on the trainval set and an error of 2e 3 on the test set using a simple MSE as our objective function.

EVALUATION

To asses the quality of our results, we conducted a blind user study. We wanted to evaluate the realism of our generated mouth animations and the impact of visual feedback on user's time perception and comprehension when listening to a potential personal assistant.

• The realism of our artificial mouth animations is evaluated by comparing them to mouths generated by a landmark tracking system applied to real videos. We considered two scenarios.

In the first one, these two video categories have to be distinguished independently. In the second one, both classes have to be discriminated by pairs. • The impact of visual feedback on user experience is evaluated through a listening and comprehension test. In this test, the user answers questions about recordings with or without mouth animations. The user must also estimate the speech's length which provides indications of its patience level.

Setup

The evaluations have been conducted in an open space environment on our experimental table presented in Figure 1. Candidates are confronted alternatively to recordings with and without the mouth animations over our different experiments presented as following.

The 24 candidates sampled for this experiment are randomly selected among English-speakers, including native ones, based on their exposure to 3D animations in the form of movies, videos games and modeling independently of their age and gender. We defined exposure as three categories: rare, casual, and daily. Finally, to avoid any bias in the results, professional animators and 3D modelers are excluded from the candidates.

Age

Exposure to 3D animations

Realism Evaluation

This evaluation estimates how our system can fool candidates. For a given audio recording, candidates need to identify and classify artificial mouth animations from tracked ones. In this sense, we confronted them to two experiments.

• Candidates are exposed to ten independent mouth animations from 3 to 6 seconds in random order and have to assign them to the appropriate category. Each candidate took 2 to 5 minutes to complete the task. In Figure 7, the median success rate among all profiles is around 50% with a standard deviation of 14.2%.

Results show that candidates are not able to correctly identify the videos as being part of one of the two classes. • To extend our experiment further and assess the robustness of the results, users are then exposed to five pairs of 3 to 6 seconds of videos from each category and have to distinguish them side by side. This task took 3 to 7 minutes to complete for each candidate. In Figure 8, candidates are still not able to discriminate the two mouth animation generation methods.

According to Figure 7, and Figure 8, gender and age do not seem to impact candidates' ability to solve the identification and classification tasks. Contrary, the exposure criteria tend to help the users. Therefore, as illustrated in Figure 9, both experiments confirm that our model can produce mouth animations realistic enough to fool human candidates.

Figure 7: Success rate during blind independent identification test of artificially generated and human tracked mouth animations. Results are displayed per independent discrimination criterion: gender (in blue), age (in green) and exposure to 3D animations (in red). Each box-plot describes the maximum, first quartile, median, third quartile and minimum of the candidates' success rate.

Comprehension Evaluation

Visual feedback is capable of affecting the human's comprehension capacity. To evaluate the impact of our mouth animations on com- prehension, we realized two experiments. Sixteen audio recordings -from 3 to 18 seconds randomly placed in 20 seconds tracks -with or without visual feedback, were shown to the user who needed to answer a comprehension question with four choices. In the first one, the user is asked to answer a question concerning the content of the speech. In the second one, an answer is shown to the user who needs to find the correct item among four possible questions, allowing to perform cross-validation on the visual feedback impact. We expect our mouth animations to alter candidates' cognition.

Results of this evaluation do not show any significant difference when visual feedback is provided or not, thus for any candidate profile.

Time Perception

Time perception can be an indicator of a user's patience during listening tests. Our generated mouth animations are expected to increase people's ability to measure this component by providing visual feedback. To this end, candidates were asked to estimate the duration of recordings during the comprehension tests. The audio recordings from the comprehension evaluation task are independently displayed one by one with or without visual feedback. Both comprehension and evaluation tasks took 7 to 10 min to complete in total.

In Figure 11, results show a small difference between candidates estimation errors with and without visual feedback. Compared to audio only, visual feedback slightly improves candidates ability to estimate the recordings' duration. For more details, we provide a histogram for each question Figure 10. These histograms show that visual feedback tends to increase candidates accuracy for the duration estimation task. Answers are less sparse and more centered around the right one than with audio only.

Interactivity Comfort

The ultimate perspective of this work is interactivity comfort with personal assistants. This contribution focuses on the interest of visual feedback during conversations between users and personal assistants. Hence, it has been challenging to evaluate how visual feedback improves interactions independently of the personal assistant's limitations. In this sense, we let candidates rate their use of a Google Assistant with and without our ADMA interface. The average rating shows 78% of the candidates estimated that comfort is significantly improved with visual feedback, independently of their gender, age or exposure to 3D animations.

Synthesis

Our proposed evaluation shows that the generated mouth animations combined to speech synthesis are realistic enough to fool, at list partially, human candidates. If initial expectations considered visual feedback as an approach to more natural HCIs, our study could not allow concluding quantitatively either qualitatively such assumptions. However, we observed that the candidates have a better estimation of the recording's length in the presence of mouth animations. Hence, we can assume that visual feedback improves time perception and comfort independently of the gender, age, and exposure to 3D animations.

CONCLUSION

In this work, we present an end-to-end Text-driven Mouth Animation model. Contributions in generative animations traditionally benefit the entertainment sector for video games and animated movies. We explore such technology to the use of virtual avatars for personal assistants. Visual feedback often enables more natural ways to interact, especially when combined with audio. Our focus in this work is the generation of text-driven mouth animations for such personal assistants to improve the user experience.

To this end, we proposed a methodology for the creation of datasets dedicated to text-driven 3D mouth animations. This approach does not require the use of performance capture and can easily be extended to future applications. We also developed an endto-end neural network model combining state of the art in speech synthesis and a custom neural regressor inspired by previous work trained on our dataset. This model allows the creation of 3D mouth animations with its associated speech audio conditioned on unique text input. We also provided an evaluation set to estimate the realism of the generated mouth animations and their value in terms of improvement for HCI. We conclude that the level of realism is good enough to fool human candidates, whereas it does not help in solving cognition task of comprehension. The interest seems limited to improving user comfort.

Possible Applications

We consider some application domains in which our model and study can be relevant for future developments.

• Video Games and animated movies could use our text-driven mouth animations to animate their characters or as placeholders for preview animations proving a better sense of the final rendering. Compared to performance capture, our system does not require the use of sophisticated software and/or hardware that some could not afford. Our proposal also provides consistent results and does not depend on the artist's animation skills. • Art museums and exhibitions could use text-driven mouth animations to renew how visitors can experience art galleries. Text-driven portraits or even text-driven description cards could bring interactivity to traditional audio guides. Such technology could be extended with other visual mouth renderings instead, the one presented in this work. • People suffering from Autism spectrum disorder (ASD) could also benefit from such technology. Controlled robotic interaction has proven to be less frustrating among these individuals and is used as a new form of therapy. Children with ASD are better at integrating audiovisual feedback compared to real people during social interactions. As our work focus on the importance of visual feedback for the creation of less frustrating natural interactions, our model could benefit those robots and enhance their ability.

Limitations

Major limitations observed in our work are directly inherited from deep learning sensitivity to the data it is trained on and its heavy computation costs.

• The visual realism fidelity of our mouth animations seems affected by the lack of horizontal movement. Our model does not seem to capture the horizontal mouth movements as good as the vertical ones. To tackle this issue, we plan to retrain our regressor network using different regularization techniques. • The computation cost of our pipeline is limited by the requirement of a powerful GPU. It cannot be suitable in some applications requiring limited physical space, power consumption, and network constraints. Hence, two aspects of the pipeline could be improved. On the one hand, WaveGlow, used in the TTS module, is a bottleneck for being able to integrate such project on Sytem On a Chip (SOC) devices and could be optimized by different approaches such as batch inference.

On the other hand, our visual rendering engine is limited by the mouth's 3D vertices and normals computation occurring on each frame and could be optimized by re-targeting the 3D landmarks on a 3D mouth model instead. • The unique voice used in our Speech Synthesizer can induce biases and is not suitable for applications requiring different gender, race or age characteristics or even neutrality. We plan to address this issue by retaining and conditioning each module of the TTS on a speaker latent variable using a multi-speaker dataset.

Future Work

One extension of this work is the introduction of an emotional and prosody context to the entire system. The emotional context of the speaker influences lips movement, speech tones, and styles. We think avatars and personal assistants should be able to express this kind of features and would result in better interactions with humans. Previous works show that a sentiment and prosody embedding or latent context variable can leverage such controls over TTS and Facial Animations independently [START_REF] Karras | Audio-driven facial animation by joint end-to-end learning of pose and emotion[END_REF][START_REF] Lee | Emotional end-to-end neural speech synthesizer[END_REF]. We want to extend our work using this approach of a learned latent variable as a way to influence the whole system in its integrity.

Figure 2 :

 2 Figure 2: The first line represents the successive 40ms Mel-spectrograms computed from a testing video audio track with the corresponding inferred 3D mouth landmarks on the second line which have been used to generate the final rendering on the last line.

Figure 3 :

 3 Figure 3: Examples of extracted landmarks in 3D coordinates using FAN algorithms on cropped faces.

Figure 4 :

 4 Figure 4: Illustration of the 20 points composing the mouth skeleton with their corresponding FAN annotations.

Figure 5 :

 5 Figure 5: 3D Spline interpolation applied on the 20 mouth landmarks. Each image represent a different interpolation resolution.From left to right and top to bottom each 3D plot respectively correspond to interpolation factors of (8, 1),[START_REF] Pham | End-to-end learning for 3d facial animation from raw waveforms of speech[END_REF][START_REF] Poria | A review of affective computing: From unimodal analysis to multimodal fusion[END_REF], (32, 8) and (64,[START_REF] Pham | End-to-end learning for 3d facial animation from raw waveforms of speech[END_REF] with horizontal interpolation first and vertical interpolation second.

Figure 6 :

 6 Figure 6: The overall architecture is composed of successive neural modules. The modules Taoctron2 and WaveGlow are in charge of the text to speech transformation. The following ADMANet module computes the 3D mouth landmarks animation based on the speech waveform. Finally, a rendering engine generates the final animation with its corresponding audio file.

Figure 8 :

 8 Figure 8: Success rate during blind side by side discrimination test of artificially generated and human tracked mouth animations. Results are displayed per independent discrimination criterion: gender (in blue), age (in green) and exposure to 3D animations (in red). Each box-plot describes the maximum, first quartile, median, third quartile and minimum of the candidates' success rate.

Figure 9 :

 9 Figure 9: Confusion matrices during blind tests for respectively independent identification (on the left) and side by side discrimination (on the right) of artificially generated and human tracked mouth animations. Each value have been transformed into percentages.

Figure 10 :

 10 Figure 10: Histograms of speech duration estimation for audio only (in blue) versus audio with visual feedback (in red). Each histogram corresponds to a different speech. Speech duration varies from 3 to 18s. Original answer for the speech duration (in green) and histogram tendency curve can be observed on the graph.

Figure 11 :

 11 Figure 11: Average error of the speech duration estimation by the candidates during comprehension test between audio only (in blue) and audio with visual feedback (in red). Each box-plot describes the maximum, first quartile, median, third quartile and minimum of the candidates' duration estimation difference from the right answer.

Table 1 :

 1 ADMA-Net architecture is composed of three stages.

		Kernel Stride	Outputs	Activation
			Frequency Domain
	Conv2D	-	-	1x64x32	-
	Conv2D	1x3	1x2	72x64x16	ReLU
	Conv2D	1x3	1x2	108x64x8	ReLU
	Conv2D	1x3	1x2	162x64x4	ReLU
	Conv2D	1x3	1x2	243x64x2	ReLU
	Conv2D	1x2	1x2	256x64x1	ReLU
			Time Domain
	Conv2D	3x1	2x1	256x32x1	ReLU
	Conv2D	3x1	2x1	256x16x1	ReLU
	Conv2D	3x1	2x1	256x8x1	ReLU
	Conv2D	3x1	2x1	256x4x1	ReLU
	Conv2D	4x1	4x1	256x1x1	ReLU
		Mouth Landmark Regression
	Dropout 0.5	-	-	256	-
	FC	-	-	256	Sigmoid
	FC	-	-	256	Sigmoid
	FC	-	-	20 ⇤ 3 = 60	Linear

Table 2 :

 2 Study candidate composition.

		18-25 26-59 Rare Casual	Daily
	Man	8	4	2	2	8
	Woman	4	8	4	6	2
	Total	12	12	6	8	10

https://doi.org/10.21785/icad2019.032


Nico Valentini
email: n.valentini@brgm.fr

Yann Balouin
email: y.balouin@brgm.fr

Assessment of a Smartphone-Based Camera System for Coastal Image Segmentation and Sargassum monitoring

Keywords: coastal video monitoring, deep learning, convolutional neural network, superpixel over-segmentation, Sargassum warning system

Coastal video monitoring has proven to be a valuable ground-based technique to investigate ocean processes. Presently, there is a growing need for automatic, technically efficient, and inexpensive solutions for image processing. Moreover, beach and coastal water quality problems are becoming significant and need attention. This study employs a methodological approach to exploit low-cost smartphone-based images for coastal image classification. The objective of this paper is to present a methodology useful for supervised classification for image semantic segmentation and its application for the development of an automatic warning system for Sargassum algae detection and monitoring. A pixel-wise convolutional neural network (CNN) has demonstrated optimal performance in the classification of natural images by using abstracted deep features. Conventional CNNs demand a great deal of resources in terms of processing time and disk space. Therefore, CNN classification with superpixels has recently become a field of interest. In this work, a CNN-based deep learning framework is proposed that combines sticky-edge adhesive superpixels. The results indicate that a cheap camera-based video monitoring system is a suitable data source for coastal image classification, with optimal accuracy in the range between 75% and 96%. Furthermore, an application of the method for an ongoing case study related to Sargassum monitoring in the French Antilles proved to be very effective for developing a warning system, aiming at evaluating floating algae and algae that had washed ashore, supporting municipalities in beach management.

Introduction

Image Classification-Coastal Area

Due to the rapid progress of ground-based and remote-sensing technology, the spatial resolution of color images and the size of image archives are being increased yearly. This has led to the development of more sophisticated and efficient algorithms and methodologies for image processing, based on up-to-date computer vision research. Aiming to investigate coastal processes, the growing need for coastal high-resolution data is being pursued. Particularly, land cover classification is presently a hot topic field.

Among the coastal video monitoring techniques, ground-sensing approaches by means of camera systems are widely used, allowing the investigation of coastal processes with high temporal and spatial resolution, with optimal accuracy. Numerous alternative or complementary uses are also pursued, with applications related to coastal hydrodynamics [START_REF] Osorio | Virtual BUOY: A video-based approach for measuring near-shore wave peak period[END_REF][START_REF] Almar | Wave runup video motion detection using the Radon Transform[END_REF][START_REF] Ibaceta | Assessing the performance of a low-cost method for video-monitoring the water surface and bed level in the swash zone of natural beaches[END_REF][START_REF] Andriolo | Nearshore Wave Transformation Domains from Video Imagery[END_REF], morphodynamics [START_REF] Bergsma | Video-based nearshore bathymetry estimation in macro-tidal environments[END_REF][START_REF] Valentini | A new video monitoring system in support of Coastal Zone Management at Apulia Region, Italy[END_REF][START_REF] Ondoa | Sea Level at the Coast from Video-Sensed Waves: Comparison to Tidal Gauges and Satellite Altimetry[END_REF], or quality monitoring (e.g., seagrass) [START_REF] Díaz-Gil | Using stereoscopic video cameras to evaluate seagrass meadows nursery function in the Mediterranean[END_REF][START_REF] Beliaeff | A general framework for indicator design and use with application to the assessment of coastal water quality and marine protected area management[END_REF]. Moreover, the use of low-cost devices and public resources has recently become a focus of attention and is being further explored [START_REF] Bracs | Evaluation of Opportunistic Shoreline Monitoring Capability Utilizing Existing[END_REF][START_REF] Harley | Shoreline change mapping using crowd-sourced smartphone images[END_REF][START_REF] Andriolo | Operational Use of Surfcam Online Streaming Images for Coastal Morphodynamic Studies[END_REF]. When dealing with coastal ground-based camera systems, with respect to satellite observations, the availability of images with smaller but optimal field-of-view and proper spatial resolution increases the opportunity of removing the barriers related to, for example, cloud cover and temporal lags, facilitating the study of large time series. As a result, this requires focus be placed on two characteristics in particular: automation and pixel-based distinctiveness.

In the literature, the algorithms employed for coastal image classification need to distinguish between classes of pixels based on a limited number of intrinsic pixel features, normally the color channels RGB/HSV/Lab, and especially focused on seeds or regions of interest [START_REF] Quartel | Object-oriented extraction of beach morphology from video images[END_REF][START_REF] Valentini | New algorithms for shoreline monitoring from coastal video systems[END_REF][START_REF] Aarninkhof | A video-based technique for mapping intertidal beach bathymetry[END_REF], or by means of a classifier which use features (e.g., geometrical, textural) that are typically discernible with the human eye (e.g., support vector machine, SVM [START_REF] Hoonhout | An automated method for semantic classification of regions in coastal images[END_REF]). In contrast to these shallow features, deep features which cannot be effectively extracted by traditional methodology have not yet played a role in coastal image classification.

Recently, deep learning has made a great deal of progress in natural image processing, as well as in remote-sensing image classification. Moreover, the performance of convolutional neural networks (CNNs) in scene classification or object detection [START_REF] Othman | Using convolutional features and a sparse autoencoder for land-use scene classification[END_REF][START_REF] Buscombe | Landscape Classification with Deep Neural Networks[END_REF] has been practically established. A CNN is a multi-layer artificial neural network with convolutional kernels, where each layer is a non-linear feature detector performing local processing of contiguous features within each layer, and it is developed by eliciting the function of the human brain [START_REF] Chen | Superpixel based land cover classification of VHR satellite image combining multi-scale CNN and scale parameter estimation[END_REF]. Presently, CNN-based land cover classification methods fall into three main categories: semantic segmentation using fully convolutional networks (FCNs) [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF], pixel-based CNN classification, and superpixel-based CNN classification, with many different configurations. The FCN is a CNN where the last fully connected layer is substituted by another convolution layer with a large "receptive field". The result is a semantically labelled image that has the same resolution as the input image. By inspecting the classification results of the FCN, it is evident that the boundaries of small-scale regions are not adequately detailed. Pixel-based CNNs are used to classify single pixels, by using their features. Applied in a pixel-based approach, a square image patch containing adjacent pixels of the pixel to be classified feeds the CNN for features extraction [START_REF] Chen | Superpixel based land cover classification of VHR satellite image combining multi-scale CNN and scale parameter estimation[END_REF]. During the training phase, training images are disaggregated into overlapping patches, and each patch is typically centered on a pixel which provides the class for the whole patch. From the literature, it is well-known that the window-sliding approach is a highly time-consuming process, especially for high-resolution images, and needs a large number of spatial processing units and hard disk space and time. Taking a high-resolution image as an example, over 2 million image patches must be fed into a CNN for pixel-level classification results [START_REF] Aggarwal | Data Classification: Algorithms and Applications[END_REF]. Traditional pixel-based CNN classification models which reach higher accuracy in classification tasks are not sufficiently resilient to image degradations, particularly against salt and pepper noise and blurring [START_REF] Roy | Effects of degradations on deep neural network architectures[END_REF]. Some strategies have recently been proposed to reduce the time spent in classifying an image through the use of superpixel segmentation, which is still challenging. Superpixels represent a form of image segmentation, but the focus lies on a controlled over-segmentation, more than in object-oriented partitioning. Segmented superpixel regions are highly homogeneous and hence are easier to be correctly classified, avoiding fragmented classification results. In fact, by adjusting their size and compactness, an image can be divided into several, almost homogeneous, regions with a certain number of pixels. Hence, it is possible to create superpixels that can be almost completely contained by a patch of determined size. Currently, many superpixel algorithms have been developed [START_REF] Wang | Superpixel segmentation: A benchmark[END_REF]. However, some are able to provide good compactness but have a bad boundary recall, such as simple linear iterative clustering (SLIC) [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF], and some have good boundary recall performance but have a bad compactness, such as the mean shift algorithm [START_REF] Lv | Very high resolution remote sensing image classification with SEEDS-CNN and scale effect analysis for superpixel CNN classification[END_REF]. It is evident that not all superpixel algorithms are suited for CNN classification since the data fed into the input layer of the CNN should consist of images with fixed sizes. Therefore, only superpixel segmentation methods with a good balance between compactness and boundary recall are suitable for CNNs.

Additionally, in order to alleviate the drawbacks related to the loss in spatial accuracy of convolutional networks, some solutions have been explored to date, e.g. by using a domain transform to simultaneously learn edge detection and semantic segmentation [START_REF] Chen | Semantic image segmentation with task-specific edge detection using CNNs and a discriminatively trained domain transform[END_REF], by including boundary detection in FCN-type models [START_REF] Marmanis | Classification with an edge: Improving semantic image segmentation with boundary detection[END_REF], or a fully connected conditional random field (CRF) algorithm [START_REF] Krähenbühl | Efficient inference in fully connected crfs with Gaussian edge potentials[END_REF] which takes into account the low-level information captured by the local interactions of pixels and edges [START_REF] Chen | Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs[END_REF].

In this work, we investigated and evaluated the possibility of using low-cost devices for the semantic segmentation of morphological/natural regions from coastal images with natural scenes. This was achieved by means of a procedure that relies on a supervised deep learning framework, which is shown to be very effective in application to pattern recognition and the semantic segmentation of not-structured, complex regions.

Sargassum Monitoring by Imagery

In the domain of coastal and water quality monitoring, the use of imagery-when visible signatures can be exploited-represents a valid alternative for collecting time series of data, which accordingly demands some efficient and easy-to-use methods for allowing automatic analysis. At present, the information retrieved from remote sensing satellite systems is very useful to the qualitative and quantitative estimation of offshore macroalgae presence and motion [START_REF] Stumpf | Applications of satellite ocean color sensors for monitoring and predicting harmful algal blooms[END_REF]. This type of investigation is significant when related to phenomena like algae washing ashore. In fact, waters off the Caribbean islands in recent years have seen large amounts of Sargassum seaweed [START_REF] Wang | Mapping and quantifying Sargassum distribution and coverage in the Central West Atlantic using MODIS observations[END_REF]. These record-breaking Sargassum blooms and mass stranding started at large in 2011, then 2015 saw the next large-scale event and in January 2018, unusually large aggregations of Sargassum were spotted on satellite imagery. Normally, when floating, offshore Sargassum provides an important habitat and refuge to a large diversity of animals. However, when weeds approach nearshore in massive quantities due to the actions of currents and winds, they start to become deathtraps for many animals and contribute to the degradation of important coastal habitats, threatening coastal activities and ecosystems. The decomposing mass, which can be several meters high, provoking olfactory, mechanical, and health problems, damages tourism activities mostly due to hydrogen sulphide, since the sight and smell make the beaches highly unappealing. Both satellite and modeled surface current data point to the North Equatorial Recirculation Region as the origin of recent mass blooms north of the mouth of the Amazon, between Brazil and West Africa, in an area not previously associated with Sargassum growth [START_REF] Gower | Satellite images suggest a new Sargassum source region in 2011[END_REF]. A number of factors, including nutrients, rising sea temperatures, and Sahara dust storms, have been put forward as potential causes [START_REF] Louime | /i> Invasion of Coastal Environments: A Growing Concern[END_REF]. Specific models developed to analyze satellite imagery and detect floating algae-the Floating Algae Index (FAI) [START_REF] Hu | A novel ocean color index to detect floating algae in the global oceans[END_REF] or the Alternate Floating Algae Index (AFAI) [START_REF] Maréchal | A simple, fast, and reliable method to predict Sargassum washing ashore in the Lesser Antilles[END_REF]-reveal that only in recent years was the area subject to mass proliferation of Sargassum: satellite imagery prior 2011 shows the area to be "largely free from seaweed".

Unfortunately, operational warning devices able to anticipate algae washing ashore still have disadvantages related to the inadequate sampling and temporal frequency (MODIS observations, e.g., [START_REF] Wang | Predicting Sargassum blooms in the Caribbean Sea from MODIS observations[END_REF]) and the interposing obstacles such as cloud shadows and sun glint constitute very important issues. Actually, there is little space for improvement, as these are natural phenomena. Aside from the need for anticipation, there is a practical need for local governments to deal with this severe issue and increase the abilities to quantify Sargassum onshore on a seasonal basis. In addition, there is a special need for management planning in order to pursue increased resilience and benefit from Sargassum influxes [START_REF] Cox | Summary Report on the Review of Draft National Sargassum Plans for Four Countries Eastern Caribbean[END_REF]. Location and amounts of Sargassum ashore should thus be accurately evaluated to properly plan their comprehensive management and anticipate beach maintenance requirements/funding.

Compared with satellite, ground-sensed imagery could provide finer geometric details and richer textural features of land covers, even if confined to coastal waters. Herein, a case study focused on the monitoring of beached and floating Sargassum aiming at the development of a warning system by exploiting automatic image segmentation is introduced and discussed.

Outline and Scope

This work explores an alternative approach for coastal image classification, aiming at addressing the aforementioned issues, which appeared to be highly effective. In this work, a strategy to reduce the time for image classification through the use of superpixel over-segmentation and CNN classification is proposed. Then, this work proposes a method for boosting the model's ability to capture fine details, by employing a fully-connected CRF. Aiming at the monitoring of floating and beached Sargassum at Martinique Island, a discussion on the implementation of the automatic segmentation of images in a warning system framework is highlighted. Finally, a discussion on the employment of a region-based majority voting strategy for further refining the segmentation map is presented.

The remainder of this paper is organized as follows: Section 2 presents the area investigated and the materials employed. Section 3 details the general framework and methods used in the experiments. Section 4 presents the details of experiments and results. The effectiveness of the proposed methodology and key issues of the superpixel-based CNN+CRF method are analyzed. Discussions about the implementation of the methodology for Sargassum warning system are presented in Section 5. Finally, Section 6 draws conclusions and summarizes future work.

Study Site and Materials

The study area (Figure 1) is situated on Martinique Island, France, which is located in the Lesser Antilles arc of the Caribbean Sea. The island is characterized by a humid tropical climate and is influenced by the trade winds throughout the year, with episodic cyclones. With respect to the coastal morphology, two-thirds of the island is constituted of sandy and vegetated (mangrove) beaches, and 75% of them are pocket beaches. A well-defined trend in shoreline retreat has been evaluated in the last 50 years [START_REF] Nachbaur | Définition des cellules sédimentaires du littoral martiniquais[END_REF]. Moreover, the above-mentioned concerns over Sargassum encroachments (Section 1.2) demand that the issue be addressed. This has led to the definition of a program for morphodynamics and algae monitoring, with the intent of implementing cost-effective, non-invasive instruments and straightforward methodologies in order to deal with the largest number of coastal sites. Therefore, a system composed of low-cost, smartphone-based cameras-SOLARCAM c (Marseille, France, https://www.solarcam.fr)-has been installed in Martinique, aiming at monitoring both coastal morphodynamics and beach/water quality. More generally, a station of this monitoring system is equipped with a camera, powered by a modified Sony Android smartphone, which produces time lapse in real time, a small solar panel (for charging), and a battery that supports the system's operation even during poor weather conditions. Up to one image every minute is transmitted over 3G, and the interval is tunable, depending on the solar panel power and especially on the scopes. A plastic case hosts the phone and a laminate band is used to affix it to a support, which could be a pole, house pylon, or even a tree. Images with a resolution of 8 megapixels are promptly stored on an FTP server. The price of the described representative station (excluding the monthly 3G communication expenses) is on the order of 400A C. The configuration guarantees good portability, cheapness, simplicity in communication and installation, and optimal resolution. Drawbacks are especially linked to network signal fluctuations, low acquisition frequency, and contingent displacements due to strong winds or temperature-related material deformations of the case or support. At the time of writing, 14 coastal sites have been equipped, starting from October 2018, at both Atlantic and Caribbean Sea sides. These include harbours and sandy/vegetated beaches. Among these, six sites were chosen for validation of the classification methodology and nine sites for presenting the calibration of the beached/floating Sargassum warning system, described in Section 5. Figure 1 shows the overall Martinique territory, with the superposed equipped coastal sites, distributed along the coastal area.

The six-camera dataset is divided into two main categories, beach and harbour. The images used for this work, (totaling 160), are divided into training and testing sets, in the respective proportions of 55% and 45%. Specifically, with respect to the beach-type category, seven classes were used, related to natural morphologies and land covers, including algae, anthropic structure, sea foam and swash, sand, sky, vegetation, and sea water. The harbour-type category does not include the sandy area, which has been found to be useless and to generate issues during the CNN retraining phase.

Methods for Data Processing

Sticky-Edge Adhesive Superpixels

In this work, over-segmentation processing is adopted by using sticky-edge adhesive superpixels [START_REF] Dollar | Fast Edge Detection Using Structured Forests[END_REF]. Data fed into the input layer of CNN classification must be images with a fixed size, whereas the segmented regions generated from traditional superpixel partitioning methods vary greatly in size and shape. The selection of the superpixel algorithm and the choice of parameters greatly affect the final results of image classification. Theoretically, the selected over-segmentation algorithm favors boundary adherence and generates superpixels with high compactness. This chosen method is built on an iterative approach motivated by SLIC [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF], but with superpixels snapped to the edges of the image, resulting in higher-quality boundaries. SLIC generates superpixels by clustering pixels based on the color similarity and proximity in the image plane. This algorithm performs k-means clustering in the 5D space of color information and pixel location, based on gradient-ascent methodology. SLIC uses the Lab color space, as it is perceptually uniform for small color distances. First, it samples k regularly spaced cluster centers. Then, each pixel in the image is attributed to the nearest cluster centre whose search area overlaps it. After assigning all pixels to cluster centers, they are updated by computing an average over the 5D vector of all pixels assigned to the cluster. This process of attributing pixels and updating centers is iteratively repeated until no new attribution occurs. Then, since the method makes use of the main image edges to snap the superpixels' region contours, structured forests is used [START_REF] Dollar | Fast Edge Detection Using Structured Forests[END_REF] to implement edge detection in the image. Structured forests is a very fast edge detector that achieves excellent accuracy, based on a structured learning approach [START_REF] Dollar | Fast Edge Detection Using Structured Forests[END_REF]. The employment of this very fast edge detector increases the boundary recall of SLIC superpixels. After the first step of superpixel partitioning, the generated superpixel-level segmented regions can be classified by using high-level features extracted by CNN.

MobileNet CNN and Mechanisms of Deep Transfer Learning

Generally speaking, a CNN is a multi-layer perceptron network with many hidden convolutional layers and pooling layers [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. It is very similar to a biological neural network. The extraction of engineered features and data recognition, necessary in the traditional artificial neural network algorithm, is avoided in CNN-type structures. This is very valuable when dealing with image processing since it is qualified in features detection including color, texture, shape, and image topology [START_REF] Zhang | Local features and kernels for classification of texture and object categories: A comprehensive study[END_REF].

MobileNet [START_REF] Howard | Efficient Convolutional Neural Networks for Mobile Vision Applications[END_REF], a recent CNN framework proposed by Google, is built on blocks called depthwise separable convolutions. Depthwise convolution applies a single filter to each input channel, followed by a pointwise convolution, which applies a 1 × 1 convolution to combine the outputs. Unlike a regular convolution which filters and combines in one step, this is called separable since it splits this step into two separate steps-one for filtering and one for combining. Particularly, in the v2 architecture (MobileNet V2 [START_REF] Sandler | MobileNetV2: Inverted Residuals and Linear Bottlenecks[END_REF]) the residual connections and the expand/projection layers are introduced. The pointwise convolution became a projection layer, which projects data with higher channels into a tensor with lower dimensions. This kind of layer is called a bottleneck layer since it normally decrements the amount of data that flows through the network. In the main block, an expansion layer is introduced, which expands the number of channels in the data which feed the depthwise convolution. Each layer has a batch normalization and the activation function is ReLU6 (except for the projection one). The novelty is also in the residual connections, which connect the beginning and end of convolutional blocks with skip connections. The network can access earlier activations which could be not modified in the convolutional block. This seems essential in order to build networks of great depth [START_REF] He | Deep Residual Learning for Image Recognition[END_REF]. The full MobileNet V2 architecture consists of 17 of these building blocks in a row, followed by a regular 1 × 1 convolution, a global average pooling layer, and a classification layer. This strategy in the convolutional blocks significantly reduces the number of parameters when compared to normal convolutions and makes the network architecture lightweight. Using these convolutions marginally sacrifices accuracy to obtain a low-complexity deep neural network, making it very suitable for mobile and embedded vision applications.

In this work, a transfer learning methodology is employed using the pre-trained CNN MobileNet-V2 architecture [START_REF] Sandler | MobileNetV2: Inverted Residuals and Linear Bottlenecks[END_REF] implemented within TensorFlow-Hub [START_REF] Nelli | Deep Learning with TensorFlow[END_REF] in order to classify coastal oblique images. As an alternative to building a model from scratch, here we explored the possibility to improve a previously trained model which had already been trained to solve a similar (larger) classification problem, specifically, on the Imagenet database [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF]. It is surprisingly effective to take advantage of features learned by a bigger database [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF]. So, a new final classifier is added to a pre-trained "frozen" model, and only the weights of this layer on top of the network are updated during training-a fully-connected layer that determines the image class given the set of features extracted from the convolutional base network. Contrary to training large models, less regularization and data augmentation techniques are used since small models have less trouble with overfitting [START_REF] Howard | Efficient Convolutional Neural Networks for Mobile Vision Applications[END_REF]. CNNs require input data to be small square image regions (patches) due to the shape limitation of the network. Since the shape of superpixels is irregular, superpixels cannot be directly used as input data to feed CNNs. Therefore, the center pixel of each superpixel is selected and a square centered on the superpixel center is used as the input data. After the classification of squares, the resolving class label is assigned to the associated superpixel. The patch images based on the superpixel centers are the data used by the MobileNet classifier, and the superpixels mine the specific pixels that are labelled in the resolving classification. Hence, the center pixel plays the role of a connection between CNN and the superpixels.

Refining via Conditional Random Field

Fully connected CRF is a probabilistic graphical model [START_REF] Krähenbühl | Efficient inference in fully connected crfs with Gaussian edge potentials[END_REF], employed in this work to smooth raw segmentation maps (the output of CNN-based superpixel classification). Each node of the graph is assumed to be linked to every other pixel in the image. Using the Gaussian CRF potentials, the method is able to consider not only neighboring information but also long-range dependencies between pixels, and contextually the model is responsive to fast mean-field inference [START_REF] Chen | Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs[END_REF]. The model couples nodes, favoring same-label assignments to spatially proximal pixels. Qualitatively, the primary function of these long-range CRFs is to clean up the spurious predictions of weak classifiers built on top of the CNN. The combination of CNN and CRF is of course not new [START_REF] Chen | Semantic Image Segmentation With Deep Con-Volutional Nets and Fully Connected CRFs[END_REF]. The approach applied here treats every pixel as a CRF node receiving unary potentials from the CNN. The model involves the use of an energy function to minimize:

E(x) = ∑ i θ i (x i ) + ∑ ij θ ij (x i , x j ), (1)
where x is the label assignment for pixels from the CNN. The unary potentials explain the data cost. They are described by θ i (x i ) =log P(x i ), where P(x i ) is the label probability at pixel i (i.e., as assigned by the CNN). The pairwise potentials, the second term of Equation ( 1), motivate the smoothness cost. They have a form that allows for efficient inference while using a fully connected graph, that is, when connecting all pairs of image pixels, i,j [START_REF] Chen | Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs[END_REF]. According to the original work in [START_REF] Krähenbühl | Efficient inference in fully connected crfs with Gaussian edge potentials[END_REF], the pairwise potential is defined as:

θ ij (x i , x j ) = µ(x i , x j )[ω 1 exp(- |p i -p j | 2 2σ 2 α - |I i -I j | 2 2σ 2 β ) + ω 2 exp(- |p i -p j | 2 2σ 2 γ )], (2)
where µ(x i , x j ), referred to as the label compatibility function, is assigned equal to 1 if x i = x j and 0 otherwise, which means that only nodes with different labels can be penalized. The expression describes two Gaussian kernels in different feature spaces, referred to as pixel compatibility function. The first, "bilateral" kernel depends on both pixel positions (identified as p) and colors (Lab space, denoted as I), and the second kernel only depends on pixel positions. The hyperparameters σ α , σ β , σ γ mainly control the scale of Gaussian kernels. The first kernel constrains pixels having similar color and position to gain similar labels, while the second kernel merely considers proximity in space when enforcing smoothness [START_REF] Chen | Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs[END_REF]. The kernel parameters ω (n) are equally weighted, and both set to 1. In particular, high-dimensional filtering algorithms [START_REF] Krähenbühl | Efficient inference in fully connected crfs with Gaussian edge potentials[END_REF] significantly speed up the computation, resulting in an algorithm that is quick in practice.

Workflow of STICKY-CNN-CRF Classification

The workflow of the superpixel-based CNN classification is shown in Figure 2, which includes two main phases-training and testing, including the application of the semantic segmentation. Firstly, by sticky-edge adhesive superpixels partitioning, the original SolarCam images from the training data are over-segmented into several regions, which are used as the spatial processing units for CNN classification. The image is manually labelled by following a modified methodology from [START_REF] Buscombe | Landscape Classification with Deep Neural Networks[END_REF]. Next, transfer learning enables the retraining of the experimental network, based on MobileNet V2 convolutional neural network, described in Section 3.4. The samples of labelled regions are used to retrain the CNN parameters, based on resolving the accuracy of the classification. The patch size used is equal to 96 pixel square. Finally, with regard to the semantic segmentation phase, the testing dataset from the original images are firstly over-segmented by means of the sticky-edge adhesive algorithm. The superpixel patches are classified using the pre-trained network. Then, the fully-connected CRF is applied for smoothing the results and better capturing the boundaries of the regions.

SolarCam images are used for superpixel partitioning, CNN training, superpixel-based classification with CNN, and finally refining via fully-connected CRF.

As an example, for the explanation of the main steps related to the semantic segmentation phase, Figure 3 shows the original image, over-segmented in superpixels (Figure 3a,d), the CNN-based superpixels classification with class labels as mentioned above (Figure 3b,e), and finally the result from the CRF refining mechanism used to smooth the segmentation map (Figure 3c,f).

Results

In this section, the performance of the method described in Section 3 is evaluated on the basis of 72 images in the testing dataset, based on six Solarcam cameras (12 images per camera), with full resolution of 3264 × 2448 pixels. Around 3000 superpixels for each image were used as processing units for CNN training.

The metric used in this work for analyzing the performance of the methodology presented is mainly based on a confusion matrix, a table with rows and columns reporting the number of false positives (FP), false negatives (FN), true positives (TP), and true negatives (TN), which allows for more detailed analysis than the mere proportion of correct classifications (accuracy). "True" or "false" indicates if the classifier predicted the class correctly, whereas "positive" and "negative" indicate if the classifier predicted the desired class. The way to summarize the table is here presented as a function of the F1-score, which is defined as an equal mean of Precision and Recall. The Precision score is based on the total number correctly classified as positive divided by the total number of predicted positives: Precision = TP TP+FP . The Recall explains the total number of correctly classified positives divided by the total number of positives: Recall = TP TP+FN . Furthermore, the sensitivity of results on the superpixels and CRF hyperparameters were assessed to demonstrate the capabilities and applications of the trained model.

Experimental Results

During the CNN transfer training procedure, the main hyperparameters were kept constant: the learning rate at 0.01, the training steps fixed as 2000. The batch size of the SolarCam classification datasets of coastal images was set to 100. The codes ran on a computer with Intel R Core TM i5-7300HQ CPU @ 2.50 GHz, 16 GB RAM, and Python 3.7. The gradient was computed via batch gradient descent, employing only the CPU. TensorFlow [START_REF] Abadi | TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems[END_REF] was chosen as the deep learning platform. Two models were trained, related to the categories mentioned above (i.e., beach morphology and harbour-type land cover). The classification accuracy was evaluated with respect to the superpixel-based CNN classification, and then by inspecting the pixel-scale results, following CRF refinement.

Table 1 summarizes the CNN prediction accuracies (F1-score) at superpixel scale. The number of training and testing superpixels was similar since the number of images used for training and testing were almost equal. The overall classification accuracy was very positive. The maximum (96%) was reached by the sky class, related to the harbour-type category, which was trained with the smaller amount of data, while the minimum was associated to the anthropic one, among both categories. Indeed, anthropic regions are often too small and difficult to detect, in both conditions. Conversely, the sky class for the beach-type category was not well predicted. It was also found that the presence of kites (Figure A1 in the Appendix A) could have invalidated the results. The algae were quite well predicted for both categories. (See Figure A1 and A2 in the Appendix A for further qualitative results).

In Figure 4, the confusion matrices describing the classification accuracies, at pixel-scale, after CNN classification and CRF detailing, are shown as normalized data per class and category under analysis. Further elucidations about the main metrics used are presented in Table 2. With respect to beach morphology, the range of classification F1-score was between 77% and 96%. The anthropic-road class reached the minimum, while the sky the maximum recall. The vegetation and anthropic-road classes had higher incorrect classifications, where a peak of 26% of FP for the vegetation class was evident. The algorithm was surprisingly able to predict beached algae well, with a 85%. As excepted, misclassification derived from recognition ambiguities with respect to spatially closer classes, particularly vegetation and sand. The foam and swash areas were combined in a unique class, considering their highly similar visual appearance and since this type of neural network has no contextual information capability. Together with the water class, they were well predicted. In the analysis of the confusion matrix related to the harbour-type category, the results appeared slightly sparse. The classification F1-score was in the range of 75% to 93%. There was often confusion between vegetation and algae classes. This should be explained because these two class types may have similar spectral features. For both categories, it is evident that the vegetation class was well predicted (high recall), but most often the buildings, which cover very tiny areas of the images, were not well recognized by the CNN classification algorithm. The path size of the CNN network plays a fundamental role in this issue since it was found that superpixels less than almost two-thirds of the path size of the CNN network (96 pixels) are not able to catch the deep features. As is well known, the overall accuracy is mainly influenced by the dominant classes in an image. A sort of salt-and-pepper effect was found at very small-scale classifications, especially for algae regions, labelled as vegetation.

Parameter Sensitivity Analysis

This paper presents a CNN-based classification method for coastal image analysis using superpixel over-segmentation and CRF refining. The following section discusses the influences of the superpixel over-segmentation input for CNN-based classification and the CRF hyperparameters, and the effects of both on the pixel-scale results.

Figure 5 shows a sensitivity analysis carried out for the evaluation of the impact of the number of superpixels used for image partitioning on the final accuracy, in which results are divided into each class studied. A higher slope was observed in the range between 20 and 200 superpixels, with respect to the right part of the curve. Approximately 850 superpixels were needed to roughly cover the patch size of 96 pixels used as the CNN processing unit, using images having 8 Mp resolution as the case study. It is clear that for almost all the classes, before reaching this value, convergence at the higher score was achieved, even if a kind of oscillation was still present, mostly attributable to the influence of CRF in the final segmentation. In order to find a compromise between computation speed and efficiency, the number of superpixel units for CNN processing in the evaluation procedure was set to 800. We also investigated the eventual influence of the compactness index in superpixel generation. This index indicates the ratio of minimum enclosing rectangle area of the superpixel to the number of pixels contained within it. After some experiments, by applying values around the default one, no specific correlation was found.

On the basis of the above analysis, it can be summarized that there was a clear scale effect in the final classification accuracy. This was particularly true in the regions with high complexity (e.g., algae). That is, the deep features appeared less valuable at large scales as there was major variability and lower consistency. There is no definite evidence to suggest that there is a unique superpixel scale for all optimal classifications in each class. The refinement step of semantic segmentation employs fully connected CRFs, which makes use of the superpixels labelled regions as unary potentials and pairwise potential terms, modeling each pair of pixels i and j in the image, no matter how far from each other they lie. The "scale" of the Gaussian kernels, which shapes the pairwise potential, is controlled by the hyperparameters (see Section 3), which significantly influence the final semantic segmentation map. By fixing the values of ω 2 = 1 and ω 1 = 1, the best combinations of the mentioned parameters were sought by exploring the solution space on a subset of the testing sample (50 images). In the literature, challenging searching solutions are employed based on schemes such as the coarse-to-fine search [START_REF] Chen | Semantic Image Segmentation With Deep Con-Volutional Nets and Fully Connected CRFs[END_REF] or structured output SVM [START_REF] Orlando | Learning fully-connected CRFs for blood vessel segmentation in retinal images[END_REF]. In this context, a calibration exercise was considered in order to estimate the optimal range of values useful for optimizing the pixel-wise classification (F1-score) accuracy. We fixed the number of mean-field iterations at 20 for the entire analysis. Furthermore, since the running time is highly influenced by the choice of the image scale, this value was fixed to 0.5, ensuring computation efficiency. Figure 6 maps the F1-score, averaged among all the classes and the two categories under analysis, in the space of the hyper-parameters σ α , σ γ , σ β . The range of values investigated was chosen considering the work in [START_REF] Chen | Semantic Image Segmentation With Deep Con-Volutional Nets and Fully Connected CRFs[END_REF]. Specifically, the ranges were σ β ∈ [2 : 12], σ α ∈ [20 : 120], and σ γ ∈ [30 : 220].

A large range of values which led to solutions having accuracies greater than 70% was found for kernel hyperparameters σ α and σ γ , while a smaller amplitude of admissible values was related to σ γ . To some extent, the former models the standard deviations for the location component of the color-dependent and color-independent kernels. The latter is related to the standard deviation for the color component of the color-dependent term, and it appeared to have a much greater impact on the acceptable accuracy. The values chosen for use in the evaluation phase presented above were

σ α = 90, σ γ = 130, σ β = 5.
Finally, as regards the superpixels image partitioning, a newly proposed and widely used algorithm, SEEDS [START_REF] Van Den Bergh | Superpixels Extracted Via Energy-Driven Sampling[END_REF], was also briefly tested in order to compare its influence on the feature extraction and final superpixel classification results with respect to the sticky-edge algorithm. The running time decreased by a factor of 1.8. With regard to the overall dataset, the superpixel classification (F1-score) achieved slightly lower (2.64%) accuracy for the final classification, obtained by averaging all types of classes. It can be concluded that sticky-edge adhesive algorithm performed slightly better than SEEDS with almost the same running time, but with benefits from the shapes of superpixel regions. The main reason is related to the fact that sticky-edge generates superpixel regions whose boundaries, derived from an edge detector, are closer to practical land cover regions. Conversely, superpixel regions generated by SEEDS could cross the border of two land covers, especially for those with blurred borders. Moreover, the non-uniformity in the size and shape of superpixel regions, more noticeable in SLIC, is supposed to improve the generalization capability of CNN, which could prevent over-fitting.

Discussion

At present, the great availability of low-cost camera systems and the great potential of portable devices, coupled with autonomous and efficient energy collectors (e.g., solar panel), push the limits of the current approaches for coastal video monitoring. Moreover, the newest advances in machine learning allow further investigation of the capabilities for image understanding. In many occasions, environmental contingencies require these types of new methodologies and tools to be easily deployable and maintainable.

Specifically, in order to face the Sargassum emergency in the Lesser Antilles and to benefit from these new methods and instruments, a warning framework has been implemented in Martinique that is useful for the quantification of floating and beached algae. The Sargassum warning framework deployed uses the two phases described in Section 3. It makes use of the models trained and presented above (two more models were trained) for employing an automatic segmentation in quasi-real-time. Presently, the SolarCam stations in Martinique collect image snapshots every hour. Every three daylight hours, starting from 09:00 a.m., an automatic routine is launched which performs an automatic image segmentation. In this scenario, a region of interest (ROI) oriented toward the field-of-view (FoV) areas prospectively affected by the beached/floating algae is considered. This allows high computation power and time during CNN classification to be avoided. The predicted classes are analyzed for the extraction of the Sargassum blobs. These potential predicted blobs are filtered based on relative euclidean distances to the other predicted classes (specifically sand, foam, terrain/vegetation, and sky, respectively) in order to remove suspected outliers (mainly false positives). Subsequently, the computation of boundaries and areas of each spot is given. Finally, the determination of an expected warning is based on site-specific temporal thresholds. Geo-referenced data are being used, where available, to report data in meters rather than pixels, by means of a standard method for georectification [START_REF] Holman | The history and technical capabilities of Argus[END_REF]. A location map where the framework is being tested is shown in Figure 7. With respect to all sites monitored, presented in Section 2 (shown as points), the map marks the warning system stations in red and green. Specifically, the green and red colors indicate the percentage of manually validated data with respect to the excluded ones, respectively. This means that a percentage is affected by different issues, mainly related to acquisition problems, loss of data, blurred conditions (i.e., the two sites facing the Caribbean Sea), foggy environment, or classification errors.

In the same Figure 7, time series of floating/beached Sargassum for a few sites as examples are drawn on top of the map. The red lines represent the temporal evolutions of the pixel/real-world areas computed every 3 h (red points) at each station, while the shaded red curves on top of the red lines, referred to as the "holes uncertainties", are computed as tiny regions inside the great algae spots, which are perceived as questionable predictions. As mentioned above, at each site under analysis, the definition of a double threshold is being realized, strictly related to spatial and temporal constraints. Particularly, as an example with respect to the beach-type category, a first absolute minimum threshold was defined, specified as a percentage of the beach covered area (30%). Furthermore, in order to ensure reliable information in the event of beaching/arrival, so as to avoid the dissemination of useless warnings (i.e., false positives), the image under analysis should exhibit a computed Sargassum area greater than 20% of the prior temporal result. The warning framework is still under validation in order to be available for communities/municipalities.

The regions predicted as algae were seen to be affected by inaccuracies, specifically inside the great blobs regions or at the boundaries, which are referred to as the questionable predictions above. The CRF algorithm adopted, based on a fully connected graph [START_REF] Krähenbühl | Efficient inference in fully connected crfs with Gaussian edge potentials[END_REF], captures long-range dependencies between image pixels. It is used to detail the CNN raw classification and to reclassify pixel labels. Its hyperparameters combination, which controls the "scale" of Gaussian kernels, was calibrated on several images with almost distinctive conditions (Section 4), but it still suffered from the aforementioned ambiguities. A straightforward way to improve the semantic segmentation output was conceived and built, based on the inclusion of the effects of boundary detection by re-employing an edge-adhesive superpixels map. So, it was surprising to investigate the dual task of the effects of superpixels on the semantic segmentation. A great interest of the coastal image community is related to the identification of the so-called precise semantic boundaries (e.g., shoreline, vegetation limit), which requires not only the detection of several boundaries but the association of semantic classes to them. In the literature, this idea is typically seen as a combination of boundary detection and semantic segmentation [START_REF] Hariharan | Semantic contours from inverse detectors[END_REF], which requires an association of precise boundaries to a semantic class related to them. A common approach to the task is to separate methods such as semantic segmentation and contour detection, and hence fuse the results of both steps. In this work, a superpixel over-segmentation was performed by employing a great number of superpixels (i.e., 50,000), allowing a further refinement and smoothing of the boundaries. So, the semantic segmentation CRF output was snapped to the superpixels by majority voting of the regions. This means that superpixels that overlapped more than 50% with the semantic class were assigned the corresponding label. Figure 8 shows an example of the result from the implementation of this further refinement. At the bottom, it highlights two magnifications extracted from a zoom-box, defined in Figure 8b. The CRF output (Figure 8c) from the methodology in Section 3 was further refined by means of the superpixels and the majority voting of the regions (Figure 8d). Particularly, it is evident that the sparse small regions labelled as water or vegetation were re-classified with the classes of the superpixels which incorporated them (mainly algae, vegetation). Moreover, it allowed a smoothing much more adherent to the reality (interfaces of water-sand or algae-sand). Particularly relevant for improving Sargassum classification, we observed an increase in classification accuracy. On the basis of 20 images from the same testing database, as above, the F1-score increased by 2.3% compared to the results discussed in Section 4 with respect to beach-type morphology. The great advantage of employing such models could be straightforward for several applications related to coastal monitoring research and case studies. Because only a few images are required for training since the superpixel partitioning creates a broad and adequate database for CNN transfer learning (i.e., the number of training superpixels in Table 1), easy prototyping of models can be realized. Applications related to shoreline detections and/or nearshore bars evolution, beach widths, dune growth, or Posidonia banquettes dynamics [START_REF] Valentini | Investigation on the capabilities of low-cost and smartphone-based coastal imagery for deriving coastal state video indicators: Applications on the upper mediterranean[END_REF] could result in effortless automation. With respect to coastal classification algorithms (e.g., based on SVM [START_REF] Hoonhout | An automated method for semantic classification of regions in coastal images[END_REF]), the transfer learning method described here allowed us to employ 10× less ground-truth data while still achieving around the same classification accuracy [START_REF] Hoonhout | An automated method for semantic classification of regions in coastal images[END_REF]. Furthermore, the powerful refining capability of the fully connected CRF makes it possible to better catch classes' borders and small scattered data, with respect to the superpixel scale [START_REF] Hoonhout | An automated method for semantic classification of regions in coastal images[END_REF].

There are still some limitations to the proposed approach, most significantly the disadvantage related to unstable plastic support of the SolarCam. This causes the cameras to experience high-frequency vibrations due to low-to-average winds, and, in a few occasions, minor absolute displacements due to wind gusts. The latter condition is very inconvenient due to loss of image geometry reference, which is useful for the real-world computation of Sargassum area. Additionally, problems of variations in lighting and weather change greatly affect distributions of color, contrast, and brightness. Herein we observed, particularly during season transitions, certain soil cover appearance changes (vegetation became reddish) and this could greatly affect the classification efficacy. Another practical limitation is due to issues in proper network coverage.

Conclusions

This work combines ideas from per-superpixel deep convolutional neural networks and fully connected CRF, creating a method able to generate semantically accurate predictions and detailed segmentation maps, while remaining computationally efficient. To achieve a high overall classification accuracy, the over-segmentation has to hold a balance between compactness and boundary accuracy. Accordingly, an algorithm based on SLIC specifically focused on snapping the boundaries, computed by means of a fast and effective edge detector, is employed. The work demonstrates the general effectiveness of a small, very fast, existing CNN framework (MobileNetV2) for image patch classification. The fully connected CRF was proved to be very efficient in detailing a sparse and raw semantic segmentation. Experimental results indicate that for beach and harbour-type image categories the average accuracies were in the range of 75% to 96%, across a dataset derived from six cameras. By means of a superpixel over-segmentation method at a variety of scales, in coastal-specific environments, the effectiveness of deep features extracted by CNN was evaluated and the optimal scale was discussed. A sensitivity of final classification accuracy to the CRF hyperparameters space is proposed. For some images with few small land-cover spots or some simple artificial regions, the methodology can suffer from ambiguities. A case study related to the implementation of a warning system for Sargassum monitoring is presented to exhibit the capabilities and potential applications of the trained models. Particularly, when a further post-processing procedure based on superpixels region-based majority voting was applied, the F1-score related to the Sargassum class reached 91%, with respect to images of beach-type morphology.

The presented procedure promotes the use of camera-based images for applications related to automatic feature recognition and coastal quality monitoring. Further experiments are being planned with a larger dataset (including other Lesser Antilles islands) and aiming at the application of the method to other sources of data, such as videos. Moreover, further studies are focused on processing the time-series data derived from the warning system in order to retrieve parametrization of the meteo-oceanographic forces useful for describing the probability of occurrence of algae degradation over time, disintegration due to wave impact, or removal due to tide oscillations.

Figure 1 .

 1 Figure 1. Martinique Island, Lesser Antilles: Distribution of the coastal sites where the Solarcam system has been deployed.

Figure 2 .Figure 3 .

 23 Figure 2. Workflow of the methodology: training phase useful for building the models by means of convolutional neural network (CNN) transfer learning mechanism, by using superpixel-based patches and the semantic segmentation phase, whose results are used for testing the pixel-scale classification accuracy. DCNN: deep CNN.

Figure 4 .

 4 Figure 4. CNN + CRF pixel-scale classification results. The ground-truth data are represented on the y-axis, the predicted ones on the x-axis. The lateral bar maps the percentage of data classified. (a) The beach category of the testing database; (b) The harbour-type category.

Figure 5 .

 5 Figure 5. The sensitivity of the final accuracy (F1-score), from CNN-based superpixels classification and CRF refinement results at the superpixel scale.

Figure 6 .

 6 Figure 6. Classification sensitivity on the fully-connected CRF hyperparameters. The σ α and σ β scales, the so-called bilateral appearance kernel, depend on the pixel positions and color, respectively. σ γ scales the Gaussian kernel, related only to pixel positions.

Figure 7 .

 7 Figure 7. Map of the locations where the SolarCam system was deployed (as in Figure 1). The points, marked in red and green , indicate the sites where the warning framework has been installed. The green and red colors pertain to the percentage of good and excluded data, respectively. As examples, five time series of Sargassum surfaces, estimated in the context of the automatic warning framework, along with holes uncertainties, are shown.

Figure 8 .

 8 Figure 8. Original image from dataset, beach-type category (a). Semantic segmentation output from CNN classification and CRF refinement. A zoom box is overlapped on the semantic segmentation map (b). At the bottom, the zoomed regions show both the magnified area (c) and the smoothed region obtained from the majority voting mechanism of the regions derived from superpixel partitioning (d).

Table 1 .

 1 Summary of superpixel-based CNN classification accuracy (F1-score), for both categories investigated: beach morphology and harbour-type land cover, respectively. Image classes were analyzed singularly. The number of training and testing superpixel patches are highlighted.

			Beach		Harbour
	Classes	F1-Score	Training Superpixels Superpixels Testing	F1-Score	Training Superpixels Superpixels Testing
	Algae	0.88	11,727	10,128	0.85	14,739	13201
	Anthro-road	0.85	339	521	0.83	2319	1563
	Foam-swash	0.88	11,814	16,230	0.93	13,026	230,217
	Sand	0.91	6864	4260	-	-	-
	Sky	0.84	1482	852	0.96	1838	1803
	Vegetation	0.89	53,676	59,426	0.87	58,321	62,032
	Water	0.95	32,679	25,362	0.93	26,449	15,260

Table 2 .

 2 Summary of results by using metrics: precision, recall, and F1-score, for both categories investigated: beach morphology and harbour-type land cover, respectively. Image classes were analyzed singularly.

			Beach			Harbour
	Classes	Precision Recall F1-Score Precision	Recall	F1-Score
	Algae	0.95	0.85	0.90	0.82	0.81	0.80
	Anthro-road	0.95	0.65	0.77	0.72	0.8	0.75
	Foam-swash	0.97	0.83	0.89	0.78	0.84	0.81
	Sand	0.80	0.89	0.84	-	-	-
	Sky	0.97	0.96	0.97	0.99	0.86	0.93
	Vegetation	0.72	0.94	0.82	0.81	0.7	0.75
	Water	0.78	0.93	0.85	0.81	0.88	0.85

J. Mar. Sci. Eng. 2020, 8, 23; doi:10.3390/jmse8010023 www.mdpi.com/journal/jmse

Acknowledgments: The video system used in this study was deployed as part of the Posidonia beach litter monitoring project funded by BRGM and DDTM 2B (Dir. Dep. Des Territoires et de la MEr de Haute-Corse), and the Sargassum monitoring network of Martinique funded by BRGM, ADEME (Ag. De L'environnement et de la maîtrise de l'Energie) and DEAL (Dir. Environnement, Aménagement et Logement) de Martinique. We particularly thank Thibaut De Cathelineau for his fundamental support in web-cam deployment. We thank the three reviewers for their constructive comments that improved the quality of the paper.

Funding: This research received external funding form the National Research Agency (ANR) through the Carnot granted project RP17MPL003.