M Porral

Laurent Krajecki

Lucas

J Randrianandrasana

A Chanonier

H Deleau

T Muller

P Porral

M Krajecki

L Lucas

Multi-user predictive rendering on remote multi-GPU clusters

Keywords: Human-centered computing-Collaborative interaction, Computer Graphics-Distributed/Network graphics, Computer Graphics-Raytracing, Predictive Rendering

Multi-user predictive rendering on remote multi-GPU clusters

INTRODUCTION

During the last two decades, virtual reality has become a promising technology widely embraced by the industry. In the automotive industry workflow, many stages including the vehicle design process, virtual prototyping and manufacturing are already benefiting from virtual reality thus resulting in a shorter time-to-market and a higher product quality [START_REF] Berg | Industry use of virtual reality in product design and manufacturing: a survey[END_REF][START_REF] Zimmermann | Virtual reality aided design. a survey of the use of vr in automotive industry[END_REF]. At the same time, the increasing need of mobility and synergy all along the industry pipeline has led to research efforts targeting delocalized collaboration [START_REF] Fleury | Remote collaboration across heterogeneous large interactive spaces[END_REF] thus opening the way to new perspectives such as networked virtual reality. However, real-time visualization tools including CAD software applications and virtual reality are still facing many challenges when the key concerns are focused on the aesthetic of the final product. Despite recent technological advances, real-time rendering can only be obtained at the price of numerous approximations on i) the underlying lighting and material models and ii) on the algorithms used to compute light-matter interactions. In practice, such visualization tools can only be used for illustrative purposes and thus, more sophisticated physically based solutions belonging to the predictive rendering field must be used when it comes to the perceived quality of the product. In contrast to the usual rendering methods, predictive rendering attempts to accurately resolve the light transport, allowing one to predict how a virtual scene would look like under the conditions defined by the virtual model [START_REF] Wilkie | Predictive rendering[END_REF].

To achieve this expected level of accuracy, predictive-oriented approaches make an exclusive use of spectrally defined materials and light inputs to prevent well-known surface metamerism * e-mail:rasolonjatovo.randrianandrasana@univ-reims.fr exhibited by traditional RGB/XYZ-based renderers. However, physically accurate results can only be guaranteed if the spectral and polarized natures of light are both carefully accounted for through unbiased rendering algorithms such as Monte Carlo path tracing. As one can expect, such a simulation requires tremendous computational resources preventing any real-time use even with cutting-edge hardware, especially when targeting high definition resolutions. As a consequence, expensive physical mockups are still preferred by decision makers when assessing the perceived quality of the vehicle.

Contribution

We propose to fill the gap between the illustrative real-time environments enabling user collaboration such as virtual reality and the computationally expensive simulations such as predictive rendering, with a predictive rendering environment taking full advantage of modern clusters to drastically reduce the rendering time. Our work mainly targets multi-GPU supercomputers frequently found in high-end industries. Additionally, the system enforces remote collaboration all along the industry pipeline by providing a remote web access enabling real-time multi-user interactions with the virtual mockup in a synchronized fashion. This paper is structured as follows: First, a review of previous work related to remote collaborative high performance graphics is given in section 2, along with the previous work related to distributed rendering. Then, section 3 describes each logical component involved in our solution as well as their relationship. The experimental settings, the results as well as the system limitations are detailed in section 4. We conclude our paper and highlight the future research directions in section 5.

RELATED WORK

Remote collaborative high-performance graphics Many research efforts addressing delocalized collaboration have been carried out. Among the previous work addressing both remote collaboration and remote high-performance graphics, Renambot et al. proposed SAGE [START_REF] Renambot | Collaborative visualization using high-resolution tiled displays[END_REF] as a scalable framework enabling collaborative visualization, mainly focused on local collaboration through tiled display systems. The framework supports distributed rendering on remote clusters and high resolution streaming to a variable number of displays. Although remote shared visualization has been partially addressed through the Visualcasting extension [START_REF] Jeong | Visualcasting-Scalable real-time image distribution in ultrahigh resolution display environments[END_REF] by Jeong, important issues such as multi-user interactions at the application level were not addressed. Repplinger et al. proposed DRONE [START_REF] Repplinger | Drone: a flexible framework for distributed rendering and display[END_REF] as a flexible framework enabling distributed physically based rendering and supporting various deployment scenarios including remote multi-user visualization and interactions. However, as the project has become outdated, the implementation does not benefit from recent advances including modern cluster interconnections such as InfiniBand and new internet standards such as HTML5 and WebRTC opening the way to ubiquitous cross-platform solutions. Commercial and free systems addressing remote visualization such as NICE DCV and VNC are also available. While these solutions can take advantage of high-performance environments, their support for collaborative scenarios remains poorly limited in terms of mutual feeling of presence and multi-user management as all users share an unique mouse cursor and any user can preempt the remote control anytime irrespectively of the ongoing interaction flow. Other remote visualization solutions such as OTOY, OnLive and PolyStream achieve low latency streaming but do not target collaboration.

Distributed rendering As the computational load may significantly vary between different image areas, one must pay a special attention to the load balancing strategy when designing a distributed rendering system. Otherwise, load imbalances can easily occur and hence penalize the system scalability. The load balancing strategies can globally be classified into three categories [START_REF] Tamm | Web-enabled Server-based and Distributed Real-time Ray-Tracing[END_REF]. Static load balancers evenly divide the image space and assign each image area to a distinct compute unit. With static balancing, the assignment layout remains the same during the entire simulation. Common approaches consist in dividing the image space into homogeneous tiles or non-contiguous pixel sets spread over the image. Such strategies can be easily implemented but usually result in load imbalances in case of homogeneous tiles [START_REF] Freisleben | Parallel raytracing: a case study on partitioning and scheduling on workstation clusters[END_REF] and in poor local performances in case of non-contiguous pixel sets [START_REF] Wald | Realtime ray tracing and its use for interactive global illumination[END_REF]. Semi-static load balancers split the image space into nonuniform tiles in order to balance their rendering costs [START_REF] Cosenza | Gpu cost estimation for load balancing in parallel ray tracing[END_REF][START_REF] Tamm | Web-enabled Server-based and Distributed Real-time Ray-Tracing[END_REF]. The layout is updated between each frame based on a high-definition timing map reporting the rendering time of tiny contiguous pixel sets. Unfortunately, the required high-definition timing is not suitable for GPU rendering as the operations are simultaneously performed on wide pixel batches. Dynamic load balancers follow a producer-consumers pattern [START_REF] Ize | Real-time ray tracer for visualizing massive models on a cluster[END_REF][START_REF] Wald | Realtime ray tracing and its use for interactive global illumination[END_REF]. The image space is divided into a set of equal size tiles much larger than the compute unit count. When rendering a new frame, each compute unit is assigned one or several tiles from the pool and request new tiles to render as it becomes idle. This process repeats until the whole frame is rendered. While these approaches result in evenly balanced workloads, the central task pool can become a bottleneck as all nodes request new tasks simultaneously hence penalizing the system scalability. As an alternative, we propose an efficient static balancing scheme taking advantage of the stochastic nature of Monte Carlo rendering.

SYSTEM DESCRIPTION

As depicted in Figure 1, the whole rendering solution is hosted on the remote cluster and each cluster node is assigned a unique role among the three component classes described below. At the core of the system, the rendering process is carried out by the compute nodes. In this purpose, each compute node runs an instance of the predictive rendering engine. The renderer consists of a spectral path tracer running on the GPU. The server acts as a facade in charge of the compute nodes coordination. This component hides the underlying distributed rendering logic by exposing only a few set of functionalities to the client application which therefore remains entirely agnostic of the actual deployment setup. The server acts as a gateway between the client and the compute nodes. Scene update commands issued by the client such as camera moves are broadcasted to all the compute nodes which in turn update the concerned scene entities on the GPU. The server is also responsible for the image assemblies and post-processing before their delivery to the client application. It is important to note that in a predictive context, the rendering of a single frame may require many seconds to minutes to obtain noise-free images, even within a high-performance environment. Thus, our system follows a progressive rendering approach to provide the user with the current simulation progress. The client is a traditional GUI application built around a central event loop. In contrast to standalone applications, the displayed image is here requested to the server at each display refresh and any interaction with the virtual scene such as camera motions and material picking trig- gers a server object update. Finally, users interact remotely with the client GUI application through the USE Together infrastructure [START_REF] Lucas | USE Together, a WebRTC-Based Solution for Multi-user Presence Desktop[END_REF]. With this middleware technology, several users can simultaneously share a native desktop environment or a specific application window from any device running a web browser and supporting the WebRTC standard. In contrast to the server and compute nodes which run on Linux OS, the client application is currently deployed on a distinct node running a Windows OS as USE Together only supports those systems at the moment. However, as USE Together is expected to support Linux OS in the near future, it will remove the unnecessary client-server communication and will allow having both streaming and multi-user management processes directly on the server.

Distributed rendering

Monte Carlo stochastic rendering is able to capture a scene image as seen from an observer by sampling light paths connecting the observer and the light sources of the scene. The final color of each pixel is then obtained by weighting the color estimates of all the rays passing through the pixel. In a predictive context though, hundreds to thousands of light paths per pixel are required to produce highquality noise-free images. Our load balancing approach exploits this specificity by distributing the light path sampling tasks of a pixel over several compute nodes. We shall now use the term compute group to refer to a set of nodes computing colorimetric estimates for the same image pixels. Following this static path-space division approach yields a naturally well-balanced workload. Moreover, as no runtime communication for task assignment is needed, work starvation is avoided, hence maximizing rendering occupancy. The pixel value is finally obtained at each display request by weighting the current estimates of all the nodes of the group. As the data traffic linearly increases with the number of participating nodes, this compositing step can significantly reduce the system reactivity. However, by combining this static path-space division approach with a static image-space division strategy, interactive compositing rates can be maintained. Thus, our system first divides the image space following an interleaved pixel scheme and then assigns each resulting partition to a distinct compute group. This partitioning step ensures a well-balanced load but can severely affect the performance of modern ray tracers which take advantages of the spatial coherence of the rays to intersect [START_REF] Wald | Realtime ray tracing and its use for interactive global illumination[END_REF]. One must therefore pay attention to the image level-of-division to apply to ensure both local rendering performance and system responsiveness. Figure 2 outlines our hybrid approach through a typical use case of the system. Each compute node starts an instance of the renderer at startup time. The client issues an initialization command to the server (1) supplying the scene to load and some setting parameters such as the rendering resolution. The image space is divided following an interleaved scheme as illustrated on the right side of Figure 2. The server broadcasts the initialization parameters to every compute node and each compute group is assigned a distinct set of pixel partitions to compute. In the depicted example, each compute node embeds two GPUs and the image-space division is defined to eight interleaved partitions. Four compute groups are then constituted, every node of a group computing estimates for the same pixels. Each node then loads the scene on the GPUs and starts an infinite rendering loop [START_REF] Cosenza | Gpu cost estimation for load balancing in parallel ray tracing[END_REF]. The initialization step is completed. The client issues an image request to the server whenever a display update is needed [START_REF] Desprat | 3devent: a framework using event-sourcing approach for 3d web-based collaborative design in p2p[END_REF]. The server forwards the request to all the groups (4) which in return send their respective pixel partitions in their current states of rendering [START_REF] Fleury | Remote collaboration across heterogeneous large interactive spaces[END_REF]. All the pixel partitions are then put together to yield the full image which is ultimately delivered and displayed on the client application [START_REF] Freisleben | Parallel raytracing: a case study on partitioning and scheduling on workstation clusters[END_REF]. Figure 3 highlights the interactions occurring within a compute group when the server is issued an image request. First, a CPU local copy of the pixel estimates residing on the GPU is done by each node of the group (4a). Then, a collective compositing pass involving all the nodes of the group is performed (4b). This compositing step results in average estimates for each pixel value of the local partitions and is implemented with the Direct-Send compositing algorithm proposed by Eilemann et al. [START_REF] Eilemann | Direct send compositing for parallel sort-last rendering[END_REF]. A careful implementation of this algorithm yields a well-balanced computing load over the whole group and partially enables communication and computing parallelization. Each node finally sends its locally weighted pixel values to the server. All the communications occurring between the server and the compute nodes are implemented with the MVA-PICH2 MPI library and therefore efficiently exploit modern network interconnections such as InfiniBand.

Collaborative web access

Lucas et al. [START_REF] Lucas | USE Together, a WebRTC-Based Solution for Multi-user Presence Desktop[END_REF] proposed USE Together as a WebRTC-based middleware enabling remote users collaboration through the web. With this infrastructure, delocalized co-worker can remotely share a native desktop environment or a specific application window, with all user interactions broadcasted in real-time. The remote video stream is continuously produced by capturing the target GUI application or the entire desktop content through native OS API calls (see Figure 4). The captured stream is therefore augmented with the different user cursor locations. In this way, all users can locate each other in the shared workspace therefore enhancing their mutual feeling of presence and the way they collaborate. Finally, the video stream is compressed before being sent to every peer using We-bRTC MediaStream dedicated channels. To that end, USE Together takes advantage of H.264 low latency encoding capabilities of modern graphics hardware by leveraging NVIDIA NVENC encoding API with its low-latency high-performance preset. User inputs are streamed through WebRTC Data channels and processed in real-time within a synchronized multi-user environment. In this environment, a user can seamlessly take the control anytime he does a specific action (mouse click and/or keyboard use) as long as the system is not already used by a third party (another user) of course. In this case, inputs other than those associated with the active user action are simply ignored. By following this approach and assuming minimum client download bandwidths of 3 Mbps (ADSL/3G/4G) for HD streaming, USE Together provides a collaborative environment through a user experience very close to a native desktop from any device running a web browser supporting the WebRTC standard.

RESULTS

The experiments were conducted with three simultaneously connected users located at hundreds of kilometers from each other with various device types (two laptops respectively equipped with an NVIDIA Geforce GT 650M and an Intel HD4000 chip and a tablet embedding a Tegra K1 chip) and through different network infrastructures (ADSL and 4G). The system used for testing was the ROMEO supercomputer from the University of Reims Champagne-Ardenne. The cluster is composed of 128 nodes embedding 2 Intel Xeon E5-2650 with 8 cores per CPU and 2 NVIDIA Tesla K20Xm GPUs. The cluster is fully interconnected with InfiniBand ConnectX-2 QDR offering a 40 Gbps theoretical network bandwidth. The scenes used for the tests are shown in Figure 5. The car exterior and interior views respectively consist of >5M and >2M triangles and around 20 physically measured materials. The car interior view particularly requires a considerable amount of rendering time as the lighting is highly indirect. Sponza is made of 260K triangles and consists of a single Lambertian material. Many indirectly illuminated areas can also be found in this scene. HDR environment maps have been used to illuminate all the scenes and the rendering resolution was set to 1920 × 1080. Table 1 shows the rendering times of the different scenes for different node counts using the proposed load balancing approach where the image-space partitioning is used up to 4 nodes. Therefore, the path-space division approach is additionally combined to the image-space division when rendering with more than 4 nodes. Note that in our benchmark setting, the rendering of an image is considered as complete when its root mean square error (RMSE) computed against a high-quality reference image has dropped below a user-specified threshold. The RMSE thresholds were respectively fixed to 0.025, 0.3 and 0.1 for the car exterior view, the car interior view and Sponza. Figure 6 reports the scaling efficiency of the load balancing scheme where the scaling efficiency is defined as the ratio of the ideal linear rendering time to the effective rendering time. As can be seen, our hybrid load-balancing strategy shows a good scalability behaviour (90% to 110%) and is able to exploit almost all computing resources we were granted access to. The slightly reduced scaling efficiency for the car exterior view can be attributed to the impact of the interleaved image-space partitioning which may vary between different scene types. Moreover, this setup achieves interactive compositing rates close to 15 frames per second on the server node, Figure Scaling efficiency of the load balancing approach. Imagespace division is used up to 4 compute nodes, then path-space division is additionally applied.

up to 64 rendering nodes. Finally, the proposed architecture results in a globally smooth experience for all the users by providing streaming rates and latencies compatible with a remote collaborative experience.

Limitations The system requires the replication of the whole scene into the GPU memory of each node which can reveal critical when dealing with large scene models. Therefore, it would be desirable to investigate on out-of-core scene management techniques to handle such cases.

CONCLUSION AND FUTURE WORK

In this paper, we have proposed a collaborative predictive rendering environment taking full advantage of the computational power offered by modern multi-GPU clusters. To make an efficient use of the remote computational resources, we have proposed a static load balancing method leveraging the stochastic nature of Monte Carlo rendering by distributing the light path sampling process. By combining this approach to a usual image-space partitioning scheme, a good scalability can be achieved with a simple static load balancing scheme which cannot be obtained with traditional static schemes. The proposed architecture not only efficiently exploits the computational resources, but also promotes co-worker mobility and collaboration by providing a remote web access allowing real-time multi-user interactions with the virtual mockup. In comparison to the usual remote visualization tools, the system emphasizes collaboration with an enhanced feeling of presence and a transparent multiple user input management. As a result, organization members can easily access the computational resources of the system and work on centralized data from any device running a web browser supporting the WebRTC standard. We are now looking forward to extend the system to cover other useful collaboration use cases. As an example, leveraging the computing resources of modern clusters to allow the remote users to work on different views of the same shared model seems to be a promising research direction [START_REF] Desprat | 3devent: a framework using event-sourcing approach for 3d web-based collaborative design in p2p[END_REF].

Figure 1 :

 1 Figure 1: Overview of the solution

Figure 2 :

 2 Figure2: The image space is divided into interleaved pixel partitions (each partition is assigned a distinct color). Each compute group is assigned a distinct set of partitions to compute.

Figure 3 :

 3 Figure 3: Collaboration of the nodes of a group during an image request

Figure 4 :

 4 Figure 4: Overview of the collaborative web infrastructure

Figure 5 :

 5 Figure 5: Scenes used for the tests: A car exterior view (left), a car interior view (middle) and Sponza (right).

Table 1 :

 1 Rendering times (seconds) of the test scenes for different node counts

	Node count	1	2	4	8	16	32	64
	Car exterior view 199.6	97.9	53.6 26.1 13.1	6.6 3.4
	Car interior view 334.3 162.9	81.7 40.4 20.3 10.1 4.9
	Sponza 461.1 220.9 112.2 55.7 28.3 13.8 7.1

ACKNOWLEDGMENTS

This work was supported by the ROMEO Computing Center and the Image Center of the University of Reims Champagne-Ardenne. The authors also thank OPEXMedia for their support on USE Together.


Kyungtae Lim
email: kyungtae.lim@ens.fr

Cheoneum Park
email: parkce@kangwon.ac.kr

Changki Lee

Thierry Poibeau
email: thierry.poibeau@ens.fr

SEx BiST: A Multi-Source Trainable Parser with Deep Contextualized Lexical Representations

We describe the SEx BiST parser (Semantically EXtended Bi-LSTM parser) developed at Lattice for the CoNLL 2018 Shared Task (Multilingual Parsing from Raw Text to Universal Dependencies). The main characteristic of our work is the encoding of three different modes of contextual information for parsing: (i) Treebank feature representations, (ii) Multilingual word representations, (iii) ELMo representations obtained via unsupervised learning from external resources. Our parser performed well in the official endto-end evaluation (73.02 LAS -4 th /26 teams, and 78.72 UAS -2 nd /26); remarkably, we achieved the best UAS scores on all the English corpora by applying the three suggested feature representations. Finally, we were also ranked 1 st at the optional event extraction task, part of the 2018 Extrinsic Parser Evaluation campaign.

Introduction

Feature representation methods are an essential element for neural dependency parsing. Methods such as Feed Forward Neural Network (FFN) [START_REF] Chen | A fast and accurate dependency parser using neural networks[END_REF] or LSTM-based word representations [START_REF] Kiperwasser | Simple and accurate dependency parsing using bidirectional LSTM feature representations[END_REF][START_REF] Ballesteros | Training with exploration improves a greedy stack lstm parser[END_REF] have been proposed to provide fine-grained token representations, and these methods provide state of the art performance. However, learning efficient feature representations is still challenging, especially for underresourced languages.

One way to cope with the lack of training data is a multilingual approach, which makes it possible to use different corpora in different languages as training data. In most cases, for instance in the CoNLL 2017 shared task [START_REF] Daniel Zeman | CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies[END_REF], the teams that have adopted this approach used a multilingual delexicalized parser (i.e. a multi-source parser trained without taking into account lexical features). However, it is evident that delexicalized parsing cannot capture contextual features that depend on the meaning of words within the sentence.

Following previous proposals promoting a model-transfer approach with lexicalized feature representations [START_REF] Guo | A representation learning framework for multi-source transfer parsing[END_REF][START_REF] Ammar | One parser, many languages[END_REF][START_REF] Lim | A system for multilingual dependency parsing based on bidirectional lstm feature representations[END_REF], we have developed the SEx BiST parser (Semantically EXtended Bi-LSTM parser), a multi-source trainable parser using three different contextualized lexical representations:

• Corpus representation: a vector representation of each training corpus.

• Multilingual word representation: a multilingual word representation obtained by the projection of several pre-trained monolingual embeddings into a unique semantic space (following a linear transformation of each embedding).

• ELMo representation: token-based representation integrating abundant contexts gathered from external resources [START_REF] Peters | Deep contextualized word representations[END_REF].

In this paper, we extend the multilingual graphbased parser proposed by [START_REF] Lim | A system for multilingual dependency parsing based on bidirectional lstm feature representations[END_REF] with the three above representations.

Our parser is open source and available at: https://github.com/CoNLL-UD-2018/ LATTICE/.

Our parser performed well in the official end-toend evaluation (73.02 LAS -4 th out of 26 teams, and 78.72 UAS -2 nd out of 26). We obtained very good results for French, English and Korean where we were able to extensively exploit the three above features (for these languages, we obtained the best UAS performance on all the treebanks, and among the best LAS performance as well). Unfortunately we were not able to exploit the same strategy for all the languages due to a lack of a GPU and, correspondingly, time for training, and also due a lack of training data for some languages.

The structure of the paper is as follows. We first describe the feature extraction and representation methods (Section 2 and 3) and then present our POS tagger and our parser based on multi-task learning (Section 4). We then give some details on our implementation (Section 5) and we finally provide an analysis of our official results (Section 6).

Deep Contextualized Token Representations

The architecture of our parser follows the multilingual LATTICE parser presented in [START_REF] Lim | A system for multilingual dependency parsing based on bidirectional lstm feature representations[END_REF], with the addition of the three feature representations presented in the introduction.

The basic token representations is as follows. Given a sentence of tokens s=(t 1 ,t 2 ,..t n ), the i th token t i can be represented by a vector x i , which is the result of the concatenation (•) of a word vector w i and a character-level vector c i of t i :

x i = c i • w i c i = Char(t i ; θ c ) w i = Word(t i ; θ w )
When the approach is monolingual, w i corresponds to the external word embeddings provided by Facebook [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF]. Otherwise we used our own multilingual strategy based on multilingual embeddings (see Section 3.2)

Character-Level Word Representation

Token t i can be decomposed as a vector of characters (ch 1 , ch 2 ,.. ch m ) where ch j is the j th character of t i . The function Char (that generates the character-level word vector c i ) corresponds to a vector obtained from the hidden state representation h j of the LSTM, with an initial state h 0 (m is the length of token t i )1 :

h j = LSTM (ch) (h 0 , (ch 1 ,ch 2 ,..ch m )) j c i = w c h m
For LSTM-based character-level representations, previous studies have shown that the last hidden layer h m represents a summary of all the information based on the input character sequences [START_REF] Shi | Combining global models for parsing universal dependencies[END_REF]. It is then possible to linearly transform this with a parameter w c so as to get the desired dimensionality. Another representation method involves applying an attention-based linear transformation of the hidden layer matrix H i , for which attention weights a i are calculated as follows:

a i = Sofmax(w att H i T ) c i = a i H i
Since we apply the Softmax function, making weights sum up to 1 after a linear transformation of H i with attention parameter w att , the selfattention weight a i intuitively corresponds to the most informational characters of token t i for parsing. Finally, by summing up the hidden state H i of each word according to its attention weights a i , we obtain our character-level word representation vector for token t i . Most recently, [START_REF] Dozat | Stanford's graph-based neural dependency parser at the conll 2017 shared task[END_REF] suggested an enhanced character-level representation based on the concatenation of h m and a i H i so as to capture both the summary and context information in one go for parsing. This is an option that could be explored in the future.

After some empirical experiments, we chose bidirectional LSTM encoders rather than a single directional one and then introduced the hidden state H i into the two-layered Multi-Layer Perceptron (MLP) without bias terms for computing the attention weight a i :

a i = Sofmax(w att2 tanh(W att1 H i T )) c i = a i H i
For training, we used the charter-level word representations for all the languages except Kazakh and Thai (see Section 5).

Corpus Representation

Following [START_REF] Lim | A system for multilingual dependency parsing based on bidirectional lstm feature representations[END_REF], we used a one-hot treebank representation strategy to encode a set of hidden state Hi is a matrix stacked on m characters. In this paper, all the letters w and W denote parameters that the system has to learn. language-specific features. In other words, each language has its own set of specific lexical features.

For languages with several training corpora (e.g., French-GSD and French-Spoken), our parser computes an additional feature vector taking into account corpus specificities at word level. Following the recent work of [START_REF] Stymne | Parser training with heterogeneous treebanks[END_REF], who proposed a similar approach for treebank representations, we chose to use a 12 dimensional vector for corpus representation. This representation tr i is concatenated with the token representation x i :

tr i = Treebank(t i ; θ tr ) x i = c i • w i • tr i
We used this approach (corpus representation) for 24 corpora, and its effectiveness will be discussed in Section 5.

Contextualized Representation

ELMo (Embedding from Language Model [START_REF] Peters | Deep contextualized word representations[END_REF]) is a function that provides a representation based on the entire input sentence. ELMo contextualized embedding is a new technique for word representation that has achieved state-of-theart performance across a wide range of language understanding tasks. This approach is able to capture both subword and contextual information. As stated in the original paper by [START_REF] Peters | Deep contextualized word representations[END_REF], the goal is to "learn a linear combination of the vectors stacked above each input word for each end task, which markedly improves performance over just using the top LSTM layer".

We trained our language model with bidirectional LSTM using ELMo as an intermediate layer in the bidirectional language model (biLM), and we used ELMo embeddings to improve again the performance of our model.

R i = {x LM i , ← → h LM i,j | = 1, ..., L} = {h LM i,j | = 0, ..., L} (1)
ELM o i = E(R i ; Θ) = γ L j=0 s j h LM i,j (2)
In ( 1), x LM i and h LM i,0 are word embedding vectors corresponding to the token layer.

← → h LM i,j is a hidden LSTM vector consisting of a multi-layer and a bidirectional LSTM layer. h LM i,j is a concatenated vector composed of x LM i and ← → h LM i,j . We computed our model with all the biLM layers weighted. In (2), s j is softmax weight that is trainable to normalize multi-layer LSTM layers. γ is the scalar parameter to efficiently train the model. We used a 1024 dimensions ELMo embedding.

Multilingual Feature Representations

The supervised, monolingual approach to parsing, based on syntactically annotated corpora, has long been the most common one. However, thanks to recent developments involving powerful word representation methods (a.k.a. word embeddings), it is now possible to develop accurate multilingual lexical models by mapping several monolingual embeddings into a single vector space. This multilingual approach to parsing has yielded encouraging results for both low- [START_REF] Guo | Cross-lingual dependency parsing based on distributed representations[END_REF] and high-resource languages [START_REF] Ammar | One parser, many languages[END_REF]. In this work, we extend the recent multilingual dependency parsing approach proposed by [START_REF] Lim | A system for multilingual dependency parsing based on bidirectional lstm feature representations[END_REF] that achieved state-of-the-art performance during the last CoNLL shared task by using multilingual embeddings mapped based on bilingual dictionaries.

Embedding Projection

There are different strategies to produce multilingual word embeddings [START_REF] Ruder | A survey of cross-lingual word embedding models[END_REF], but a very efficient one consists in simply projecting one word embedding on top of the other to make both representations share the same semantic space [START_REF] Artetxe | Learning principled bilingual mappings of word embeddings while preserving monolingual invariance[END_REF]. The alternative involves directly generating bilingual word embeddings from bilingual corpora (Gouws et al., 2015;[START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF], but this requires a large amount of bilingual data aligned at sentence or document level. This kind of resource is not available for most language pairs, especially for underresourced languages.

We thus chose to train independently monolingual word embeddings and then map these word embeddings one to another. This approach is powerful since monolingual word embeddings generally share a similar structure (especially if they have been trained on similar corpora) and so can be superimposed with little information loss.

To project embeddings, we applied the linear transformation method using bilingual dictionar-ies proposed by [START_REF] Artetxe | Learning bilingual word embeddings with (almost) no bilingual data[END_REF]. We took the bilingual dictionaries from OPUS2 and Wikipedia.

The projection method can be described as follows. Let X and Y be the source and target word embedding matrix so that x i refers to i th word embedding of X and y j refers to j th word embedding of Y. And let D be a binary matrix where D ij = 1, if x i and y j are aligned. Our goal is to find a transformation matrix W such that Wx approximates y. This is done by minimizing the sum of squared errors:

arg min W m i=1 n j=1 D ij x i W -y i 2
The method is relatively simple since converting a bilingual dictionary into D is quite straightforward. The size of the dictionary used for training is around 250 pairs, and the projected word embedding is around 1.8GB. The dictionaries and the projected word embeddings are publicly available on Github.3

Training with Multilingual Embedding

After having trained multilingual embeddings, we associate them with word representation w i as follows:

w i = Word(t i ; θ mw )
We applied the multilingual embedding mostly to train the nine low-resource languages of the 2018 CoNLL evaluation, for which only a handful of annotated sentences were provided.

Multi-Task Learning for Tagging and Parsing

In this section, we describe our Part-Of-Speech (POS) tagger and dependency parser using the encoded token representation x i based on Multi-Task Learning (MTL) [START_REF] Zhang | A survey on multitask learning[END_REF].

Part-Of-Speech Tagger

As presented in Section 2 and 3, our parser is based on models trained with a combination of features, encoding different contextual information. However, the attention mechanism for the character-level word vector c i is focusing only on a limited number of features within the token, and the word representation element w i is thus needed to transform a bidirectional LSTM, as a way to capture the overall context of a sentence. Finally, a token is encoded as a vector g i :

g i = BiLSTM (pos) (g 0 , (x 1 ,x 2 ,..x n )) i
We transform the token vector g i to a vector of the desired dimensionality by two-layered MLP with a bias term to classify the best candidate of universal part-of-speech (UPOS):

p i = W pos2 leaky relu(W pos1 g i T ) + b pos y i = arg max j p ij
Finally, we randomly initialize the UPOS embedding as p i and map the predicted UPOS y i as a POS vector:

p i = Pos(y i ; θ pos )

Dependency Parser

To take into account the predicted POS vector on the main target task (i.e. parsing), we concatenate the predicted POS vector p i with the word representation w i and then we encode the resulting vector via BiLSTM. This enriches the syntactic representations of the token by back-propagation during training:

v i = BiLSTM (dep) (v 0 , (x 1 ,x 2 ,..x n )) i
Following [START_REF] Dozat | Deep biaffine attention for neural dependency parsing[END_REF], we used a deep bi-affine classifier to score all the possible head and modifier pairs Y = (h,m). We then selected the best dependency graph based on Eisner's algorithm [START_REF] Eisner | Efficient parsing for bilexical context-free grammars and head automaton grammars[END_REF]. This algorithm tries to find the maximum spanning tree among all the possible graphs:

arg max valid Y (h,m)∈Y Score M ST (h, m)
With this algorithm, it has been observed that parsing results (for some sentences) can have multiple roots, which is not a desirable feature. We thus followed an empirical method that selects a unique root based on the word order of the sentence, as already proposed by [START_REF] Lim | A system for multilingual dependency parsing based on bidirectional lstm feature representations[END_REF] to ensure tree well-formedness. After the selection of the best-scored tree, another bi-affine classifier is applied for the classification of relation labels, based on the predicted tree. We trained our tagger and parser simultaneously using a single objective function with penalized terms: loss = αCrossEntropy(p , p (gold) ) + βCrossEntropy(arc , arc (gold) ) + γCrossEntropy(dep , dep (gold) )

where arc and dep refer to the predicted arc (head) and dependency (modifier) results.

Since UAS directly affects LAS, we assumed that UAS would be crucial for parsing unseen corpora such as Finnish PUD, as well as other corpora from low-resource languages. Therefore, we gave more weight to the parameters predicting arc than rel and p , since arc directly affects UAS. We set α = 0.1, β = 0.7 and γ = 0.2. Unfortunately, during the testing phase, we did not adjust weight parameters that would have benefited LAS for the 61 big treebanks, and this made our results on big treebanks suffer a bit (7 th ) compared to those we obtained on Small and PUD treebanks (3 th ) regarding LAS. This also explains the gap between the UAS and LAS scores in our overall results.

Implementation Details

In this section, we provide some details on our implementation for the CoNLL 2018 shared task (Zeman et al., 2018b).

Training

We have trained both monolingual and multilingual models for parsing. In the first case, we simply used the available Universal Dependency 2.2 corpora for training (Zeman et al., 2018a). In the second case, for the multilingual approach, as both multilingual word embeddings and corresponding training corpora (in the Universal Dependency 2.2 format) were required, we concatenated the corresponding available Universal Dependency 2.2 corpora to artificially create multilingual training corpora.

The number of epochs was set to 200, with one epoch processing the entire training corpus in each language and with a batch size of 32. We then picked the best five performing models to parse the test corpora on TIRA [START_REF] Potthast | Improving the reproducibility of PAN's shared tasks: Plagiarism detection, author identification, and author profiling[END_REF].

The five models were used as an ensemble run (described in Section 5.2).

Hyperparameters. Each deep learning parser has a number of hyperparameters that can boost the overall performance of the system. In our implementation, most hyperparameter settings were identical to [START_REF] Dozat | Stanford's graph-based neural dependency parser at the conll 2017 shared task[END_REF], except of course those concerning the additional features we have introduced before. We used 100 dimensional character-level word representations with a 200 dimensional MLP, as presented in Section 2, and for corpus representation, we used a 12 dimensional vector. We set the learning-rate to 0.002 with Adam optimization.

Multilingual Embeddings. As described in Section 3, we specifically trained multilingual embedding models for nine low-resource languages. Table 2 gives the list of languages for which we adopted this approach, along with the language used for knowledge transfer. We selected language pairs based on previous studies [START_REF] Lim | A system for multilingual dependency parsing based on bidirectional lstm feature representations[END_REF][START_REF] Lim | Multilingual Dependency Parsing for Low-Resource Languages: Case Studies on North Saami and Komi-Zyrian[END_REF][START_REF] Partanen | Dependency parsing of code-switching data with cross-lingual feature representations[END_REF] for bxr, kk, kmr, sme, and hsb, and the others where chosen based on the public availability of bilingual dictionaries (this explains why we chose to map several languages with English, even when there was no real linguistically motivated reason to do so). Since we could not find any pre-trained embeddings for pcm nsc, we applied a delexicalized parsing approach based on an English monolingual model.

ELMo. We used ELMo weights to train specific models for five languages: Korean, French, English, Japanese and Chinese. ELMo weights were pre-trained using the CoNLL resources provided4 . We used AllenNLP5 for training, and used the default hyperparameters. We included ELMo only at the level of the input layer for both training and inference (we set up dropout to 0.5 and used 1024 dimensions for the ELMo embedding layer in our model). All the other hyper-parameters are the same as for our other models (without ELMo).

Testing

All the tests were done on the TIRA platform provided by the shared task organizers. During the test phase, we applied an ensemble mechanism using five models trained with two different "seeds". Python random library and are used to initialize the two parameters W and w (see Section 2). Generally, an ensemble mechanism combines the best performing models obtained from different seeds, so as to ensure robustness and efficiency. In our case, due to a lack of a GPU, different models have been trained simply based on the use of two different seeds. Finally, the five best performing models produced by the two seeds were put together to form the ensemble model. This improved the performances by up to 0.6%, but other improvements could be expected by testing with a larger set of seeds.

Hardware Resources

The training process for all the language models with the ensemble and ELMo was done using 32 CPUs and 7 GPUs (Geforce 1080Ti) in approximately two weeks. The memory usage of each model depends on the size of external word embeddings (3GB RAM by default plus the amount needed for loading the external embeddings). In the testing phase on the TIRA platform, we submitted our models separately, since testing with a model trained with ELMo takes around three hours. Testing took 46.2 hours for the 82 corpora using 16 CPUs and 16GB RAM.

Results

In this section, we discuss the results of our system and the relative contributions of the different features to the global results.

Overall results. The official evaluation results are given in Table1. Our system achieved 73.02 LAS (4 th out of 26 teams) and 78.71 UAS (2 nd out of 26).

The comparison of our results with those obtained by other teams shows that there is room for improvement regarding preprocessing. For example, our system is 0.86 points below HIT-SCIR (Harbin) for sentence segmentation and 1.03 for tokenization (HIT-SCIR obtained the best overall results). Those two preprocessing tasks (sentence segmentation and tokenization) affect tagging and parsing performance directly. As a result, our parser ranked second on small treebanks (LAS), where most teams used the default segmenter and tokenizer, avoiding the differences on this aspect. In contrast, we achieved 7 th on the big treebanks, probably because there is a more significant gap (1.72) here at the tokenization level.

Corpus Representation. Results with corpus representation (corpora marked tr in column Method of Table 1) exhibit relatively better performance than those without it, since tr makes it possible to capture corpus-oriented features. Results were positive not only for small treebanks (e.g., cs fictree and ru taiga) but also for big treebanks (e.g., cs cac and ru syntagrus). Corpus representation with ELMo shows the best performance for parsing English and French.

Multilinguality. As described in Section 3, we applied the multilingual approach to most of the low-resource languages. The best result is obtained for hy armtdp, while sme giella and hsb ufal also gave satisfactory results. We only applied the delexicalized approach to pcm nsc since we could not find any pre-trained embeddings for this language. We got a relatively poor result for pcm nsc, despite testing different strategies and different feature combinations (we assume that the English model is not fit for it).

Additionally, we found that character-level representation is not always helpful, even in the case of some low-resource languages. When we tested kk ktb (Kazakh) trained with a Turkish corpus, with multilingual word embeddings and characterlevel representations, the performance dramatically decreased. We suspect this has to do with the writing systems (Arabic versus Latin), but this theory should be further investigated.

sme giella is another exceptional case since we chose to use a multilingual model trained with three different languages. Although Russian and Finnish do not use the same writing system, applying character and corpus representation improve the results. This is because the size of the training corpus for sme giella is around 900 sentences, which seems to be enough to capture its main characteristics.

Language Model (ELMo). We used ELMo embeddings for five languages: Korean, French, English, Japanese and Chinese (they are marked with el in the method column in Table 1). The experiments with ELMo models showed excellent overall performance. All the English corpora, fr gsd and fr sequoia in French, and Korean ko kaist obtained the best UAS. We also obtained the best LAS for English en gum and en pud, and for fr sequoia in French.

Contributions of the Different System Components to the General results. To analyze the effect of the proposed representation methods on parsing, we evaluated four different models with different components. We set our baseline model with a token representation as

x i = w i • c i • p i ,
where w i is a randomly initialized word vector, c i is a character-level word vector and p i is a POS vector predicted by UDpipe1.1 (note that we did not apply our 2018 POS tagger here, since it is trained jointly with the parser and that affects the overall feature representation). We then initialized the word vector w i with external word embeddings as provided by the CoNLL shared organizers. We also re-run the experiment by adding treebank and ELMo representations. The results are shown in Table 3 (em denotes the use of the external word embedding and tr and el denotes treebank and ELMo representations respectively.). We observe that each representation improves the overall results. This is especially true regarding LAS when using ELMo (el), which means this representation has a positive effect on relation labeling.

Extrinsic Parser Evaluation (EPE 2018). Participants in the CoNLL shared task were invited to also participate in the 2018 Extrinsic Parser Evaluation (EPE) campaign6 [START_REF] Fares | The 2018 Shared Task on Extrinsic Parser Evaluation. On the downstream utility of English Universal Dependency parsers[END_REF], as a way to confirm the applicability of the developed methods on practical tasks. Three downstream tasks were proposed this year in the EPE: biomedical event extraction, negation resolution and opinion analysis (and each task was run independently from the others). For this evaluation, participants were only required to send a parsed version of the different corpora received as input back to the organizers using a UD-type format (the organizers then ran the different scripts related to the different tasks and computed the corresponding results). We trained one single English model for the three tasks using the three English corpora provided (en lines, en ewt, en gum) without treebank embeddings (tr), since we did not know which corpus embedding would perform better. In addition, we did not apply our ensemble process on TIRA since it would have been too time consuming.

Our results are listed in Table 4. They include an intrinsic evaluation (overall performance of the parser on the different corpora considered as a whole) [START_REF] Nivre | Universal dependency evaluation[END_REF] and taskspecific evaluations (i.e. results for the three different tasks). In the intrinsic evaluation, we obtained the best LAS among all the participating systems, which confirms the portability of our approach across different domains. As for the taskspecific evaluations, we obtained the best result for event extraction, but our parser did not perform so well on negation resolution and opinion analysis. This means that specific developments would be required to properly address the two tasks under consideration, taking semantics into consideration.

Conclusion

In this paper, we described the SEx BiST parser (Semantically EXtended Bi-LSTM parser) developed at Lattice for the CoNLL 2018 Shared Task. Our system was an extention of our 2017 parser [START_REF] Lim | A system for multilingual dependency parsing based on bidirectional lstm feature representations[END_REF] with three deep contextual representations (multilingual word representation, corpus representations, ELMo representation). It also included a multi-task learning process able to simultaneously handle tagging and parsing. SEx BiST achieved 73.02 LAS (4 th over 26 teams), and 78.72 UAS (2 nd out of 26), over the 82 test corpora of the evaluation. In the future, we hope to improve our sentence segmenter and our tokenizer since this seems to be the most obvious target for improvements to our system. The generalization of ELMo representation to new languages (beyond what we could do for the 2018 evaluation) should also have a positive effect on the results.

Table 1 :

 1 The seeds are integers randomly produced by the Official experiment results for each corpus, where tr (Treebank), mu (Multilingual) and el (ELMo) in the column Method denote the feature representation methods used (see Section 2 and 3).

		Corpus		UAS	LAS Rank(UAS) Rank(LAS) Baseline(LAS)
		Overall (82)		78.71 73.02	2	4	65.80
		Big treebanks only (61)	85.36 80.97	4	7	74.14
		PUD treebanks only (5)	76.81 72.34	3	3	66.63
		Small treebanks only (7) 75.67 68.12	2	3	55.01
		Low-resource only (9)	37.03 23.39	4	5	17.17
	Corpus	Method UAS(Rank) LAS(Rank)	Corpus	Method UAS(Rank) LAS(Rank)
	af afribooms		87.42 (7)	83.72 (8)	it isdt	tr	92.41 (6)	89.96 (8)
	grc perseus	tr	79.15 (4)	71.63 (8)	it postwita	tr	77.52 (6)	72.66 (7)
	grc proiel	tr	79.53 (5)	74.46 (8)	ja gsd	tr, el	76.4 (6)	74.82 (6)
	ar padt		75.96 (8)	71.13 (10)	ja modern		29.36 (8)	22.71 (8)
	hy armtdp	tr, mu	53.56 (1)	37.01 (1)	kk ktb	tr, mu	39.24 (15)	23.97 (9)
	eu bdt		85.72 (7)	81.13 (8)	ko gsd	tr, el	88.03 (2)	84.31 (2)
	br keb	tr, mu	43.78 (3)	23.65 (5)	ko kaist	tr, el	88.92 (1)	86.32 (4)
	bg btb		92.1 (9)	88.02 (11)	kmr mg	tr, mu	38.64 (3)	27.94 (4)
	bxr bdt	tr, mu	36.89 (3)	17.16 (4)	la ittb	tr	87.88 (8)	84.72 (8)
	ca ancora		92.83 (6)	89.56 (9)	la perseus	tr	75.6 (3)	64.96 (3)
	hr set		90.18 (8)	84.67 (9)	la proiel	tr	73.97 (6)	67.73 (8)
	cs cac	tr	93.43 (2)		91 (2)	lv lvtb	tr	82.99 (8)	76.91 (11)
	cs fictree	tr	94.78 (1)	91.62 (3)	pcm nsc	tr, mu	18.15 (21)	11.63 (18)
	cs pdt	tr	92.73 (2)	90.13 (7)	sme giella	tr, mu	76.66 (1)	69.87 (1)
	cs pud	tr	89.49 (7)	83.88 (9)	no bokmaal		91.4 (5)	88.43 (11)
	da ddt		85.36 (8)	80.49 (11)	no nynorsk	tr	90.78 (8)	87.8 (11)
	nl alpino	tr	90.59 (2)	86.13 (5)	no nynorsklia	tr	76.17 (2)	68.71 (2)
	nl lassysmall	tr	87.83 (2)	84.02 (4)	cu proiel		77.49 (6)	70.48 (8)
	en ewt	tr, el	86.9 (1)	84.02 (2)	fro srcmf		91.35 (5)	85.51 (7)
	en gum	tr, el	88.57 (1)	85.05 (1)	fa seraji		89.1 (7)	84.8 (10)
	en lines	tr, el	86.01 (1)	81.44 (2)	pl lfg	tr	95.69 (8)	92.86 (11)
	en pud	tr, el	90.83 (1)	87.89 (1)	pl sz	tr	92.24 (9)	88.95 (10)
	et edt		86.25 (7)	82.33 (7)	pt bosque		89.77 (5)	86.84 (7)
	fo oft	tr, mu	48.64 (9)	25.17 (17)	ro rrt		89.8 (8)	84.33 (10)
	fi ftb	tr	89.74 (4)	86.54 (6)	ru syntagrus	tr	93.1 (4)	91.14 (6)
	fi pud	tr	90.91 (4)	88.12 (6)	ru taiga	tr	79.77 (1)	74 (2)
	fi tdt	tr	88.39 (6)	85.42 (7)	sr set		90.48 (10)	85.74 (11)
	fr gsd	tr, el	89.5 (1)	86.17 (3)	sk snk		86.81 (11)	82.4 (11)
	fr sequoia	tr, el	91.81 (1)	89.89 (1)	sl ssj	tr	87.18 (10)	84.68 (10)
	fr spoken	tr, el	79.47 (2)	73.62 (3)	sl sst	tr	63.64 (3)	57.07 (3)
	gl ctg	tr	84.05 (7)	80.63 (10)	es ancora		91.81 (6)	89.25 (7)
	gl treegal	tr	78.71 (2)	73.13 (3)	sv lines	tr	85.65 (4)	80.88 (6)
	de gsd		82.09 (8)	76.86 (11)	sv pud	tr	83.44 (3)	79.1 (4)
	got proiel		73 (6)	65.3 (8)	sv talbanken	tr	89.02 (4)	85.24 (7)
	el gdt		89.29 (8)	86.02 (11)	th pud	tr, mu	0.33 (21)	0.12 (21)
	he htb		66.54 (9)	62.29 (9)	tr imst		69.06 (7)	60.9 (11)
	hi hdtb		94.44 (8)	90.4 (12)	uk iu		85.36 (10)	81.33 (9)
	hu szeged		80.49 (8)	74.21 (10)	hsb ufal	tr, mu	54.01 (2)	43.83 (2)
	zh gsd	tr, el	71.48 (5)	68.09 (5)	ur udtb		87.4 (7)	80.74 (10)
	id gsd		85.03 (3)	77.61 (10)	ug udt		75.11 (6)	62.25 (9)
	ga idt		79.13 (2)	69.1 (4)	vi vtb		49.65 (6)	43.31 (8)
	Corpus	Projected languages UAS LAS
	hy armntdp		Greek	1	1
	br keb		English	3	5
	bxr bdt		Russian	3	4
	fo oft		English	9	17
	kk ktb		Turkish	15	9
	kmr mg		English	3	4
	pcm nsc		-	21	18
	sme giella	Finnish+Russian	1	1
	th giella		English	21	21
	hsb ufal		Polish	2	2
					148

Table 2 :

 2 Languages trained with multilingual word embeddings and their ranking.

	Representation Methods UAS LAS
	baseline	81.79 78.45
	+em	83.39 80.15
	+em, tr	83.67 80.64
	+em, el	85.47 82.72
	+em, tr, el	85.49 82.93

Table 3 :

 3 Relative contribution of the different representation methods on the overall results.

Table 4 :

 4 Official evaluation results on three EPE task (see https://goo.gl/3Fmjke).

Note that i refers to the i th token in the sentence and that j refers to the j th character of the i th token. Here, we use lowercase italics for vectors and uppercase italics for matrices. So

http://opus.nlpl.eu/

https://github.com/jujbob/ multilingual-models

http://hdl.handle.net/11234/1-1989

https://github.com/allenai/allennlp

http://epe.nlpl.eu/

Acknowledgments

KyungTae Lim is supported by the ANR ERA-NET ATLANTIS project. This work is also supported by the LAKME project funded by IDEX PSL (ANR-10-IDEX-0001-02). Lastly, we want to thank the National Science & Technology Information (NTIS) for providing computing resources and the Korean national R&D corpora.


Yaroub Elloumi

Ben Mbarek

Rahma Boukadida

Mohamed Akil

Mohamed Hedi Bedoui

Fast and Accurate Mobile-Aided Screening System of Moderate Diabetic Retinopathy

Keywords: Fundus image, diabetic retinopathy, hard exudates, SVM, mobile-aided screening system

The Diabetic Retinopathy (DR) is a worldwide eye disease that causes visual damages and can leads to blindness. Therefore, the detection of the DR in the early stages is highly recommended. However, a delay is registered for ensuring early DR diagnosis which caused by the low-rate of the ophthalmologists, the deficiency of diagnosis equipment and the lack of mobility of elderly patients.

In this paper, the main objective is to provide a mobile-aided screening system of moderate DR. Within this aim, we propose a classifier-based method which is based on detecting the Hard Exudate (HE) lesions that occur in moderate DR stage. A set of features are extracted to ensure an accurate and robust detection with respect to modest quality of fundus images. Moreover, the detection is provided in a low complexity processing to be suitable for mobile device. The aimed system corresponds to the implementation of the method on a smartphone associated to an optical lens for capturing fundus image. The system reached satisfactory screening performance where an accuracy of 98.36%, a sensitivity of 100% and specificity of 96.45% are registered using the DIARETDB1 fundus image databases. Moreover, the screening is performed in an average execution time of 2.68 seconds.

INTRODUCTION

The Diabetic Retinopathy (DR) is an eye abnormality caused by long-term diabetes which it represents the leading cause of blindness of middle-aged and elderly people [START_REF] Tan | Automated segmentation of exudates, haemorrhages, microaneurysms using single convolutional neural network[END_REF][START_REF] Bastawrous | The global inverse care law: a distorted map of blindness[END_REF]. The DR prevalence is increasing at an alarming rate in world-wide where the number of DR-patient has been estimated to rise 126.6 million in 2030 [START_REF] Zheng | The worldwide epidemic of diabetic retinopathy[END_REF]. The DR progresses through 4 stages what are mild, moderate, sever and proliferative DR [START_REF] Kaur | A generalized method for the segmentation of exudates from pathological retinal fundus images[END_REF]. In the mild stage, the micro-aneurysm lesions appear as small red dots caused by the dilation of thin vessel. This stage does not require referring to an ophthalmologist unless Diabetic Macular Edema is joined [START_REF]Evidence-based Clinical Practice Guideline[END_REF][START_REF]Grading diabetic retinopathy from stereoscopic color fundus photographs-an extension of the modified Airlie House classification: ETDRS report number 10. Early Treatment Diabetic Retinopathy Study Research roup[END_REF]. The moderate DR is deduced when Hard Exudate "HE" lesions [START_REF] Amin | A method for the detection and classification of diabetic retinopathy using structural predictors of bright lesions[END_REF] crop up, which are formed by lipids and proteins leaked from blood vessels. The HE appears as bright yellow clusters of varying shapes, sizes and locations. their sizes can vary from very few pixels to a large lesion as an Optic Disk (OD). Patients in moderate DRstage have a risk up to 27% to achieve proliferative stage within one year [START_REF]Evidence-based Clinical Practice Guideline[END_REF][START_REF]Grading diabetic retinopathy from stereoscopic color fundus photographs-an extension of the modified Airlie House classification: ETDRS report number 10. Early Treatment Diabetic Retinopathy Study Research roup[END_REF]. For that reason, the detection of the DR in the early stages is highly recommended.

However, a delay is registered for ensuring early DR diagnosis caused by: the low-rate of the ophthalmologists [START_REF] Resnikoff | The number of ophthalmologists in practice and training worldwide: a growing gap despite more than 200,000 practitioners[END_REF]; the deficiency of diagnosis equipment; and the lack of mobility of elderly patients. In order to solve those problems, many automatic detection methods were proposed for early DR diagnosis, particularly for HE detection that indicates at least reaching the moderate DR [START_REF] Kaur | A generalized method for the segmentation of exudates from pathological retinal fundus images[END_REF][START_REF] Zhou | Automatic Detection of Exudates in Digital Color Fundus Images Using Superpixel Multi-Feature Classification[END_REF][START_REF] Long | Automatic Detection of Hard Exudates in Color Retinal Images Using Dynamic Threshold and SVM Classification: Algorithm Development and Evaluation[END_REF][START_REF] Adem | Exudate detection for diabetic retinopathy with circular Hough transformation and convolutional neural networks[END_REF][START_REF] Fraz | Multiscale segmentation of exudates in retinal images using contextual cues and ensemble classification[END_REF][START_REF] Harangi | Automatic exudate detection by fusing multiple active contours and regionwise classification[END_REF]. In the last few years, several optical lens have been put forward in order to capture the fundus images using the smartphone camera [START_REF] Panwar | Fundus Photography in the 21st Century--A Review of Recent Technological Advances and Their Implications for Worldwide Healthcare[END_REF][START_REF] Bolster | How the smartphone is driving the eye-health imaging revolution[END_REF][START_REF] Russo | Comparison of Smartphone Ophthalmoscopy With Slit-Lamp Biomicroscopy for Grading Vertical Cup-to-Disc Ratio[END_REF]. Those gadgets are well known with their low prices and the modest quality of captured images [START_REF] Micheletti | Current and Next Generation Portable Screening Devices for Diabetic Retinopathy[END_REF][START_REF] Darma | The quality and usability of smartphone and hand-held fundus photography, compared to standard fundus photography[END_REF]. In fact, many works proved that ophthalmologists are able to predict the presence of the different DR lesions from smartphone captured fundus images. The clinical study described in [START_REF] Ryan | Comparison Among Methods of Retinopathy Assessment (CAMRA) Study: Smartphone, Nonmydriatic, and Mydriatic Photography[END_REF] has shown a sensitivity of 95% and specificity of 94% when detecting the DR lesions on fundus images taken by mobile smartphone. In [START_REF] Rajalakshmi | Validation of Smartphone Based Retinal Photography for Diabetic Retinopathy Screening[END_REF], the ophthalmologists were detected DR with 92.7 % of sensitivity, 98.4 % of specificity and 0.90 of Kappa (k) using the mobile phone (FOP) camera. Moreover, the D-eye lens was allowed distinguishing the HEs with a Kappa coefficient between 0.822 and 0.878 [START_REF] Muiesan | Ocular fundus photography with a smartphone device in acute hypertension[END_REF].

Therefore, the challenge is to propose an automated method for DR lesion detection that: (1) ensure a higher performance detection; (2) even with the moderate quality of smartphone captured fundus images; (3) and having a lower complexity to provide result in a reduced execution time. The implementation of the method on a smartphone associated with an optical lens for capturing fundus image relies to a mobile-aided screening system. The proposed system will be able to be used by non-ophthalmologist doctor to detect the DR. Based on it mobility and low price, such system presents an adequate solution to remedy the problems related to the delay on DR screening.

Within this objective, several state-of-the-art methods were proposed. The work described in [START_REF] Recep | Deep Learning Frameworks for Diabetic Retinopathy Detection with Smartphone-based Retinal Imaging Systems[END_REF] addressed the problems of limited field of view and quality of smartphone captured fundus images. Hence, an automated method was proposed to ensure a higher performance DR grading even with blurred classical fundus images. The execution time was evaluated only when method was run in desktop. Our first work described in [START_REF] Majumder | A deep learning-based smartphone app for real-time detection of five stages of Diabetic Retinopathy[END_REF] was aimed to provide DR grading of classical fundus images when an overall accuracy of 87.4% was achieved. The method was implemented in smartphone where classification takes between 200 and 250 milli-seconds. In [START_REF] Saeed | An Approach to Automatic Hard Exudate Detection in Retina Color Images by a Telemedicine System Based on the d-Eye Sensor and Image Processing Algorithms[END_REF], a method of HE detection is propounded where a trained model was stored on cloud platform. This method was performed a higher performance detection with an accuracy of 98% using a private classical image dataset, despite providing the decision into one minute and 15 seconds.

In this paper, we propose a novel automated method for moderate DR screening achieving a better trade-off between HE detection rate and the detection time. The method is based on classifier-based method for high performant locating of HE in fundus images. In addition, the processing has a low complexity to be suitable for mobile implementation. The proposed method is implemented into a smartphone to provide an accurate and fast mobile-aided screening system for moderate DR. Our paper is organized as follow. In section 2, we present the different processing steps of our method. In section 3, we describe the implementation of the proposed method to provide the mobile-aided screening system. The experimentation is depicted in section 4 where achieved results in terms of detection performance and execution time are detailed. The last section is dedicated to the discussion and the conclusion.

THE PROPOSED METHOD FOR HARD EXUDATE DETECTION

Having a fundus image as an input, the method pipeline is composed by three main processing. The first step consists in removing the OD and partitioning the retina into several candidate regions. Thereafter, a set of features are extracted for each region which reflect the HE characteristics. The third step brings to classify regions as clear or containing HE lesions.

Pre-processing

Optic disk location

The HEs are considered as bright regions in a way that their intensities are closed to the OD one. Moreover, the lipid leakage involves a yellow color similar to the color of optic nerve fibers. Therefore, this similarity leads to increase false positive shapes when detecting the HEs. Accordingly, it is essential to remove OD, as done in several HE detection methods [START_REF] Banerjee | Detection of hard exudates using mean shift and normalized cut method[END_REF][START_REF] Banerjee | Detection of hard exudates using 2D Otsu algorithm in digital retinal fundus image[END_REF][START_REF] Zhou | Automatic Detection of Exudates in Digital Color Fundus Images Using Superpixel Multi-Feature Classification[END_REF][START_REF] Kusakunniran | Hard exudates segmentation based on learned initial seeds and iterative graph cut[END_REF][START_REF] Wisaeng | Improved fuzzy C-means clustering in the process of exudates detection using mathematical morphology[END_REF]. The method proposed in [START_REF] Shahri | Computationally efficient optic nerve head detection in retinal fundus images[END_REF] insures locating the OD based on circularity and intensity through applying the radon transform. The method achieved a higher accurate performance among the existing methods [START_REF] Haleem | Automatic extraction of retinal features from colour retinal images for glaucoma diagnosis: a review[END_REF][START_REF] Ben Sayadia | Computational Efficiency of Optic Disk Detection on Fundus Image: A survey[END_REF], where an accuracy of 100% is performed to locate OD in the public DRIVE database. We note that this method is able to be configured with the aim of locating a sub-image having the same OD size. Thereafter, this method is extended to be provide a mobile computer aided system for OD detection [START_REF] Elloumi Y | mobile computer aided system for optic nerve head detection[END_REF]. The processing was optimized to reduce complexity where the same detection accuracy is performed through 1.3 second in smartphone device. Therefore, the method proposed by [START_REF] Elloumi Y | mobile computer aided system for optic nerve head detection[END_REF] will be implemented which extract the OD sub-image [START_REF] Ben Sayadia | Computational Efficiency of Optic Disk Detection on Fundus Image: A survey[END_REF] in order to remove it from the fundus image.

Fundus image partition

Several methods have proceeded to browse all image pixels, which bring to false positive detection caused by noise pixels. Moreover, such processing leads to higher computational complexity methods. With respect to the objective of DR moderate screening, we proceed to assess the image by region instead by pixels, in order to rise the accuracy, the robustness and to enhance the computational efficiency. For such need, fundus image is partitioned into region based on retinal shapes in order to check the HE existence on each region separately.

The partition is ensured by applying the super-pixel algorithm [START_REF] Van Den Bergh | SEEDS: Superpixels Extracted Via Energy-Driven Sampling[END_REF][START_REF] Stutz | Superpixels: An Evaluation of the State-of-the-Art[END_REF] which recognized by its feasibility, robustness and low complexity [START_REF] Zhengqin | Superpixel segmentation using Linear Spectral Clustering[END_REF] where regions are defined based on color and shape borders. The LSC super-pixel algorithm spreads the fundus image while trying to minimize regions by providing them with nearest areas. Knowing that HEs may occur as small distributed lesions, this algorithm dispatches HE lesions into different regions, as illustrated in the fundus image of Fig. 1 (a) which is partitioned into 462 regions. The SLIC super-pixel algorithm partitions iteratively the fundus image until all region pixels have similar morphologies. However, it involves muffled small HE lesions with relatively wide area, as indicated depicted in fundus image of Fig. 1 (b) despite that it is partitioned into 612 regions. The SEEDS super-pixel algorithm provides better partitioning and succeeds to extract each HE cluster in a single region [START_REF] Bergh | Depth SEEDS: Recovering incomplete depth data using superpixels[END_REF] without exceed 616 regions. Moreover, retinal background with stationary color and shape are extracted in single wide region, as shown in Fig. 1 (c). Accordingly, we proceed to employ the SEEDS super-pixel algorithm.

Feature extraction

The HE lesions have several morphological proprieties where the main ones are the higher intensity, the yellow color and the sharp border. the proposed optical lenses allow light leakage when capturing fundus images. Therefore, it involves brightness noise shapes that prevents optimal detection of HE lesions. Elsewhere, the handheld aspect brings to provide a blurred fundus image which inhibits modelling explicitly the HE shapes. To surpass those limitations, we proceed to extract three features reflecting respectively intensity, color and sharp border, as described in the following sub-sections.

Sharp border feature

The HEs arise due to lipid leakage from blood vessel, which remain bundled in the retina. Therefore, the HE lesions are characterized by sharp borders, in contrast to the others DR lesions [START_REF] Sánchez | Retinal image analysis based on mixture models to detect hard exudates[END_REF]. Accordingly, we proceed to generate a feature that reflexes the sharp edge of HEs. Among several edge detection methods [START_REF] Gupta | Sobel Edge Detection Algorithm[END_REF], we employ the Sobel edge detector [START_REF] Vijayakumari | Exudates Detection Methods in Retinal Images Using Image Processing Techniques[END_REF] which has a smoothing effect on the random noise in the image. The Sobel edge filter is applied by calculating a corresponding Gradient Magnitude (GM) of the fundus image to get the edge image as output [START_REF] Vincent | A Descriptive Algorithm for Sobel Image Edge Detection[END_REF], as described in eq.( 1).

( , ) = ⃗ ( , ) = + (1)
Where and are the derivatives in horizontal and vertical directions, using respectively the mask

-1 0 1 -2 0 2 -1 0 1
and the mask

-1 -2 -1 0 0 0 1 2 1
to each pixel. Our method consists of providing a single feature that reflect the HE sharp edge. Within this objective, we compute the means edge value of each candidate region as indicated in eq.( 2).

= * ∑ ∑ ( , ) (2)
where is the result of the sobel edge filter and * is the sub-image size. The figure 2 (b) corresponds to the result of the sobel edge filter where the HE borders are illustrated explicitly modelled. Similarly, the retina background is depicted with lower intensities since it is characterized by a smoothed texture. Indeed, the region containing the HE lesions which are depicted with red borders incorporate the edges detected through the sobel filter, as shown in Fig. 2 (f).

Brightness feature

The leaked of lipids and proteins derived from the retinal blood vessels [START_REF] Kaur | A generalized method for the segmentation of exudates from pathological retinal fundus images[END_REF] appear as bright yellow crystalline granules in the retinal image. Therefore, the HEs are considered as bright lesions having higher intensity values [START_REF] Joshi | A review on exudates detection methods for diabetic retinopathy[END_REF], compared to the other retinal components such as blood vessels, fovea, and red lesions such as haemorrhage and micro-aneurysm [START_REF] Zhou | Automated Detection of Red Lesions Using Superpixel Multichannel Multifeature[END_REF]. For that reason, we aim to extract a feature that reflects the brightness of HEs. In fact, HE lesions appear as "whiter" gray level than most of the rest of the image [START_REF] Harwood | Parallelisation of an interactive lattice-Boltzmann method on an Androidpowered mobile device[END_REF][START_REF] Lahmiri | Automated detection of circinate exudates in retina digital images using empirical mode decomposition and the entropy and uniformity of the intrinsic mode functions[END_REF][START_REF] García | Detection of Hard Exudates in Retinal Images Using a Radial Basis Function Classifier[END_REF]. Therefore, we proceed to extract the gray-scale channel to model HE as brightness shapes in the retinal images, as shown in Fig. 2 (c). Thereafter, the mean gray value of each candidate region is calculated as described in eq.( 3).

= * ∑ ∑ ( , ) (3)

Color feature

The HE lesions spread upward the retina with dense composition. Therefore, there are characterized by yellowish color without a red shade, contrary to the retina background, the vessel tree and the others DR lesions. Moreover, the HE color remain the same whatever the HE sizes, locations and shapes [START_REF] Fang | Automatic segmentation of hard exudates in fundus images based on boosted soft segmentation[END_REF] [START_REF] Rajan | An Analytical Method for the Detection of Exudates in Retinal Images Using Invertible Orientation Scores[END_REF]. For that reason, we aim to extract a feature that reflects the color of HE. For this purpose, the fundus image is converted from RGB space to the HSV color space in order to select the Hue channels which is defined as the ratio of the red and the green value of each pixel [START_REF] Pa | Identification of hard exudates in retinal images[END_REF]. Consequently, the HE lesions appear as dark shapes in relation to the other retinal components, such has illustrated in Fig. 2 (e). Thereafter, our method consists in computing the means value of the color features of each candidate region as expressed in eq.( 4).

= * ∑ ∑ ( , ) (4)
Where is the hue image and * is the sub-image size.

Classification

In this section, the objective is to choose a classifier that ensures a high performance classification through the extracted features. In addition, the training should be ensured with a small dataset size due to the lack a smartphone captured fundus image database. Besides, the classifier should have a lower complexity to provide result on a reduced execution time. Within this Objective, we select three classifiers that are the Support Vector Machine (SVM), the Random Forest (RF) and the K-Nearest Neighbors (KNN). The result of each classifier depend on how the parameters are initialized. For that, each classifier is experimentally employed with different values of parameters, as described in section 4.1.2. Hence, the classifier that achieves better performances is selected to be employed on our method.

MOBILE-AIDED SCREENING SYSTEM OF MODERATE DIABETIC RETINOPATHY

The method is implemented as an app in an android mobile device, using the android studio editor where machine learning and image processing are performed through the Open computer vision (OpenCV) library [START_REF] Hudelist | OpenCV Performance Measurements on Mobile Devices[END_REF][START_REF] Bradski | Learning OpenCV: Computer Vision with OpenCV Library[END_REF]. This library contains over than 2500 optimized functions which is used in many areas such as medical imaging and security. In addition, the OpenCV library also includes a general-purpose Machine Learning Library (MLL) [START_REF] Bradski | Learning OpenCV: Computer Vision with OpenCV Library[END_REF]. To create our Android app, an access to the operating system components is needed which is assured by a high-level Application Programming Interface (API) available in JAVA language [START_REF] Harwood | Parallelisation of an interactive lattice-Boltzmann method on an Androidpowered mobile device[END_REF]. The method is implemented in the application layer as indicated in Fig. 3. The functions related to the image processing, training and testing the classifier were called through the JAVA Native Interface (JNI) combined with the OpenCV library. The access to the OpenCV library could be done either using a direct JNI call or through Java API following, as modelled by arrows in Fig. 3. The developed user interface of the mobile app is shown in Fig. 4, which contain buttons for loading and testing a fundus image. Similarly, the HE detection result and the execution time are illustrated are shown above the selected image.

EXPERIMENTAL RESULTS

Evaluation of the HE detection method

Databases and evaluation metrics

For the evaluation experiment, the DiaretDB1 database is selected which is commonly used by automated methods for DR screening, where some fundus images are illustrated in Fig 5 . It contains 89 fundus images having a size of 1500 × 1152 captured by digital camera with a 50-degree field of view. The HE lesions have been manually located by four independent medical experts. The performance of our application will be measured based on image-based criterion: if fundus image contains at least one HE regions, this image is considered "at least in moderate DR stage". Otherwise, it considered as "healthy or in mild DR stage" [START_REF] Jelinek | Automated Image Detection of Retinal Pathology[END_REF][START_REF] Osareh | A Computational-Intelligence-Based Approach for Detection of Exudates in Diabetic Retinopathy Images[END_REF]. We aim to evaluate the ability of our method to identify correctly each class. Within this objective, the performance will be measured through the accuracy, the sensitivity and the specificity metrics which are computed as described eq.( 5), eq.( 6) and eq.( 7). Accuracy =

( ) (5) Sensitivity
= ( ) (6) Specificity
= ( ) (7)
Where TP (True Positive) is the number of the abnormal images correctly classified, FP (False Positive) is the number of normal images wrongly classifier as abnormal, TN (True Negative) normal images correctly classified as healthy and FN (False Negative) is the number of abnormal images wrongly classified as normal.

Performance evaluation of the HE detection method

The goal of our method is to detect the HE presence on regions, separately. With this aim, the superpixel algorithm is applied to the fundus images, where 160 regions are selected from each database to perform the evaluation. Thereafter, the feature set is extracted for each region. Thereafter, the three classifiers are trained using the input features. the SVM is tested separately with the sigmoid, the polynomial and the RBF kernels, while the C, γ and d parameters are fixed using the grid search method [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF]. For the RF classifier, the forest trees are tested with 6, 8 and 10 depths while feature number and tree are fixed to three and 500. The KNN classifier is trained where cluster size K is varied following the above values {5, 7, 9}. The performance metrics of each classifier is show in Table 1, where we deduce that using the SVM with the polynomial kernel allows achieving the highest detection performance with an accuracy of a sensitivity 100% and a specificity of 96.45%. In addition, we deduce that the detection performance remains highly, despite of the different classifier and their parameter, which attests that extracted features reflect the HE criteria.

Healthy images

Fundus image with HEs

Evaluation with respect to existing methods

To ensure a fair evaluation, the provided performance metrics are compared to the ones achieved be existing methods using the same DiaretDB1 dataset, as depicted in Table 2. Our method outperforms the detection performance of the methods proposed in [START_REF] Kaur | A generalized method for the segmentation of exudates from pathological retinal fundus images[END_REF][START_REF] Zhou | Automatic Detection of Exudates in Digital Color Fundus Images Using Superpixel Multi-Feature Classification[END_REF][START_REF] Fraz | Multiscale segmentation of exudates in retinal images using contextual cues and ensemble classification[END_REF][START_REF] Harangi | Automatic exudate detection by fusing multiple active contours and regionwise classification[END_REF]. It achieves a higher accuracy than the ones recently proposed in [START_REF] Kaur | A generalized method for the segmentation of exudates from pathological retinal fundus images[END_REF][START_REF] Long | Automatic Detection of Hard Exudates in Color Retinal Images Using Dynamic Threshold and SVM Classification: Algorithm Development and Evaluation[END_REF][START_REF] Fraz | Multiscale segmentation of exudates in retinal images using contextual cues and ensemble classification[END_REF][START_REF] Harangi | Automatic exudate detection by fusing multiple active contours and regionwise classification[END_REF]. Moreover, our method registers the highest sensitivity compared to all methods. This result is caused by the adequate choice of the HE properties and the corresponding features. Accordingly, the achieved performance is promising for diagnosis through smartphone captured images.

Complexity analysis of the proposed method

The OD location method described in [START_REF] Elloumi Y | mobile computer aided system for optic nerve head detection[END_REF] consists at applying the radon transform to a fundus image with a size of ( × ) in order to model intensity. Thereafter, a radon space is extracted for each sub-image in order to locate OD based on pic of intensities. Therefore, the OD location requires ( ²) to be performed. The SEEDS superpixel algorithm leads to partition fundus image into regions and refine iteratively the regions boundary, which have a complexity of ( . ( )) [START_REF] Zhang | A Simple Algorithm of Superpixel Segmentation With Boundary Constraint[END_REF]. Each feature leads to explore pixels of each region to provide the feature value. Knowing that the whole image is partitioned into regions, each feature and hence the whole feature extraction step requires ( ²) de be done. The SVM testing consists at injecting features in the trained model to predict the classification [START_REF] Ayinala | Low complexity algorithm for seizure prediction using Adaboost[END_REF]. Therefore, a constant time complexity equal to (1) is required. Accordingly, the whole proposed method for HE detection has a complexity of ( ²).

The execution time evaluation

The implementation is run into the smartphone "OnePlus 5" where hardware and software parameters are detailed in Table 3. Table Therefore, the execution time of our implementation is compared to state-of-the art methods where their computational performances are shown in Table 4. Aside the method proposed in [START_REF] Saeed | An Approach to Automatic Hard Exudate Detection in Retina Color Images by a Telemedicine System Based on the d-Eye Sensor and Image Processing Algorithms[END_REF], we note that the indicated methods are always implemented and executed using MATLAB software tool in desktop architectures which have wide powerful processing compared to the mobile device. Despite, our method involves the fastest HE detection compared to the indicated works.

HE detection methods

Execution time (in seconds) L. Giancardo 2012 [START_REF] Giancardo | Exudate-based diabetic macular edema detection in fundus images using publicly available datasets[END_REF] 9.3 X. Zhang 2014 [START_REF] Zhang | Exudate detection in color retinal images for mass screening of diabetic retinopathy[END_REF] 35 I. N. Figueiredo 2015 [START_REF] Figueiredo | Automated lesion detectors in retinal fundus images[END_REF] 15.97 A. R. G. Harwood 2017 [START_REF] Harwood | Parallelisation of an interactive lattice-Boltzmann method on an Androidpowered mobile device[END_REF] 240 J. H. Tan 2017 [START_REF] Tan | Automated segmentation of exudates, haemorrhages, microaneurysms using single convolutional neural network[END_REF] From 3,4 to 4,1 W. Zhou 2017 [START_REF] Zhou | Automatic Detection of Exudates in Digital Color Fundus Images Using Superpixel Multi-Feature Classification[END_REF] 10 ~ 19 J. Kaur 2018 [START_REF] Kaur | A generalized method for the segmentation of exudates from pathological retinal fundus images[END_REF] 9.36 E. Saeed [START_REF] Saeed | An Approach to Automatic Hard Exudate Detection in Retina Color Images by a Telemedicine System Based on the d-Eye Sensor and Image Processing Algorithms[END_REF] 75 Proposed method 2.68

CONCLUSION

The (DR) requires to be diagnosed in early stages. The lack of ophthalmologist and imaging equipment cause a delay on early DR diagnosis, and hence contribute to the wide number of patients having damaged vision caused by DR. In this paper, we have proposed a mobile-aided screening system for moderate DR based on detecting the HE lesions. The experimental results allow to deduce a satisfactory performance of DR screening and prove the robustness of the method

Execution time

Average execution time against modest quality of fundus images. Moreover, the low complexity of the method processing involves performing screening in reduced execution time which is suitable for clinical context.

In near future, we aim to enhance performance of the proposed system where HE location can be improved taking into account their neighborhood of blood vessel tree and DR red lesions. the preprocessing can also be extended to rectify blur and noise that occur due to the handheld aspect of smartphone capturing. Moreover, the system can be extended to provide a DR classification whatever the stage is, through detection all DR lesions. The classifier is able to be replace by a deep learning architecture, as the one proposed in [START_REF] Majumder | A deep learning-based smartphone app for real-time detection of five stages of Diabetic Retinopathy[END_REF]. Elsewhere, the timing performance can be improved by taking benefit from the parallelism offered by recent smartphone architectures.

Figure 1 .

 1 Figure 1. Super-pixel results: (a) the LSC algorithm; (b) the SLIC algorithm; (c) the SEEDS algorithm

Figure 2 .

 2 Figure 2. HE candidates in sub-image regions: (a) the original sub-image, (b) the sobel edge filter result; (c) the gray channel; (d) the yellow channel; (e-h) the result of applying the super-pixel algorithm of the sub-images (a-d)

Figure 3 .

 3 Figure 3. Illustration of the connection between the developed application, the Java API and the OpenCV library

Figure 4 .

 4 Figure 4. The user interface of the mobile-aided screening system

Figure 5 .

 5 Figure 5. healthy and DR-affected fundus images from DIARETDB1 database

Table 1 .

 1 HE Detection performance in terms of classifiers and classifier parameters

	Classifier		SVM			RF			KNN
		Sigmoid	RBF	Poly	depth = 6	8	10	5	7	9
	Acc (%)	88.57	97.95	98.36	97.75	97.55	96.53	95.91	93.87	95.91
	Sens (%)	86.82	98.32	100	97.16	96.50	96.42	100	96.15	100
	Spec (%)	90.92	97.67	96.45	98.12	98.46	96.39	92	91.30	92

Table 2 .

 2 Comparative results for image-based evaluation for the proposed method

	Method	Accuracy	Sensitivity	Specificity
	B. Harangi 2014 [62]	82	86	--
	M. Fraz 2017 [61]	87	92	81
	W. Zhou 2017 [9]	---	87.56	94.65
	J. Kaur 2018 [2]	87±4.29	91±3.34	94±2.28
	K. Adem 2018 [60]	98.53	99.2	97.97
	S. Long 2019 [59]	97.7	97.5	97.8
	Proposed method	98.36	100	96.45

Evaluation of the mobile CAD system for HE detection

Table 3 .

 3 The technical sheets of the "OnePlus 5" Smartphone

	component	Characteristics
	OS	Android 7.1.1 Nougat
	Chipset	Qualcomm MSM8998 snapdragon 835 (10 nm)
	GPU	Adreno 540
	Storage & memory	64GB, 6 Gb RAM
	Figure 6. execution time of the mobile-aided screening system for 25 EyePACS fundus images

Table 4 .

 4 Comparison of execution time with existing HE detection methods

Authors' background