Christos Makris
email: makri@ceid.upatras.gr

Dionisios Pettas
email: petta@ceid.upatras.gr

Georgios Pispirigos
email: pispirig@ceid.upatras.gr

Distributed Community Prediction for Social Graphs based on Louvain algorithm

Keywords: Community Detection, Machine Learning, Distributed Computing, Supervised Learning, Louvain algorithm

Nowadays, the problem of community detection has become more and more challenging. With application in a wide range of fields such as sociology, digital marketing, bio-informatics, chemical engineering and computer science, the need for scalable and efficient solutions is strongly underlined. Especially, in the rapidly developed and widespread area of social media where the size of the corresponding networks exceeds the hundreds of millions of vertices in the average case. However, the standard sequential algorithms applications have practically proven not only infeasible but also terribly unscalable due to the excessive computation demands and the overdone resources prerequisites. Therefore, the introduction of compatible distributed machine learning solutions seems the most promising option to tackle this NP-hard class problem. The purpose of this work is to propose a novel distributed community detection methodology, based on the supervised community prediction concept that is extremely scalable, remarkably efficient and circumvent the intrinsic adversities of classic community detection approaches.

Introduction

In recent years, the amount of available information continuously grows, making the need for efficient information representation and data compactness more and more crucial. Therefore, the adoption of data structures such as graphs have frequently been the case, in order to conveniently enhance and readily model the primary information with its corresponding relationship structure. Characteristically, many sectors such as biology, chemistry, sociology, marketing and computer science have embraced information networks since they seem extremely suitable for hierarchical data representation [START_REF] Fortunato | Community detection in graphs[END_REF]. Especially, in targeted marketing and personalized recommendations, where the concept of retrieving the underlying hierarchical structure from complex networks have been a major concern. Hence, community detection has become one of the most significant and concurrently challenging problems in graph analysis, having as principal goal the identification of subgroups that contain highly similar entities by only using topological information. The entities similarity is calculated using specific graph topology criteria such as the edge/node connectivity degree, the clustering coefficient, the edge betweenness centrality etc. [START_REF] Basuchowdhuri | Fast Detection of Community Structures using Graph Traversal in Social Networks[END_REF]. The returned subgroups, also known as communities, should be typically densely intra-connected and sparsely inter-connected with each other.

Focusing on the social media, their information can be exceptionally modeled using graphs, where each node could represent a unique user and the connection between two users could generally express any kind of social interaction. Due to the natural human tendency to get associated and mainly interact with peers of similar interests, the formation of virtual clusters and communities can be considered as a doubtless consequence [1] [20]. Therefore, the community detection in social networks can be interpreted as the identification of groups of social media users that are either directly or indirectly connected and tend to interact more often with each other comparing to users of different communities [START_REF] Held | Dynamic Clustering in Social Networks using Louvain and Infomap Method[END_REF]. The concept of finding like-minded user subgroups has indeed proven beneficial in e-commerce, social media marketing and recommendation services.

Profuse measures have been proposed for the retrieved community structure evaluation [1] [20]. Among all, the most broadly used is the modularity which is basically a function that quantifies the average connectivity degree of a given community [START_REF] Fortunato | Community detection in graphs[END_REF]. Intuitively, modularity seeks to reflect the concentration of remaining edges within communities, after community detection application, compared to a random distribution of connections on the original graph [START_REF] Satu | Graph clustering[END_REF]. The modularity values lie between -1 and 1, where positive modularity values reveal a well-identified / well-assessed community structure and vice versa the negative ones indicate a much more complicated subjacent community structure that have not been fully identified yet. Apart from being an indispensable clustering evaluation benchmark, modularity also serves as the elementary method for graph clustering optimization in the great majority of existing methods and algorithms [3] [7] [10] [START_REF] Wang | High-performance social networking: microblog community detection based on efficient interactive characteristic clustering[END_REF] [13] [START_REF] Mark | Michelle Girvan: Finding and Evaluating Community Structure in Networks[END_REF]. Nonetheless, despite its broad use, modularity optimization is inherently classified as a NP-complete problem [START_REF] Fortunato | Community detection in graphs[END_REF] with typical implementations of high polynomial computational complexity. Apart from the undisputed modularity maximization computational difficulties, the actual size of real-world social networks acts as another major inhibitory factor. It is a common fact that today's social networks have outreached the billions of user profiles, with Facebook reporting more than 2 billion users in 2018 while the corresponding number for Instagram exceeded the 800 million users. Consequently, for social graphs of such extend, it has been obviously infeasible to even traverse through the entire network using the classic methodologies. Hence, it seems utterly pointless to even try to apply the standard graph clustering approaches described in [START_REF] Fortunato | Community detection in graphs[END_REF] [2] [4] [5] [START_REF] Mark | Michelle Girvan: Finding and Evaluating Community Structure in Networks[END_REF] since the required calculations outstrip the linear complexity. Thereupon, only scalable distributed solutions that make use of the topological properties retrieved from a small but representative part of the original graph, could rise to the occasions. As a result, considering that link prediction techniques could be very prolific for tremendously big information networks [START_REF] Amato | How Data Mining and Machine Learning Evolved from Relational Data Base to Data Science, A Comprehensive Guide Through the Italian Database Research Over the Last 25 Years[END_REF] [9], the community detection problem can alternatively be interpreted as the probability calculation problem of nodes belonging on the same community for each directly connected pair of nodes.

Inspired by the aforementioned idea, a novel distributed methodology, which follows the idea of link prediction, is proposed. Given that Louvain algorithm's [START_REF] Basuchowdhuri | Fast Detection of Community Structures using Graph Traversal in Social Networks[END_REF] performance is generally considered as the golden standard in graph clustering, a network feature analysis of a manageable but representative subgraph is initially performed. The main objective of this analysis is to classify each of the edges included as either "community" or "non-community". Obviously, this edge labelling procedure transforms this subgraph to a supervised machine learning-ready dataset that will be given as training input to a distributed logistic regression model. This trained community prediction model will be finally applied in terms of the original graph's community structure identification. The experimentation with various real-world social networks verifies that the proposed method, is remarkably promising, highly scalable and noticeably efficient for community detection despite slightly sacrificing part of original Louvain's accuracy. The remainder of this manuscript is organized as follows:

• In Section 2 the related work on community detection methodologies is given.

• In Section 3 the proposed methodology and its implementation is comprehensively presented. • In Section 4 the experimentation results are discussed.

• In Section 5 the conclusions and the future work are provided.

Related Work

The community detection significance has attracted the interest from various scientific fields. Copious methods from different backgrounds and objectives have been proposed to detect the underlying community structure of a given graph by leveraging its topological information. There is a plethora of algorithms and methodologies [START_REF] Fortunato | Community detection in graphs[END_REF] [2] [START_REF] Bisma | Network Community Detection: A Review and Visual Survey[END_REF], which can be roughly distinguished as divisive algorithms, agglomerative algorithms, transformation methods and other approaches accordingly.

Divisive Algorithms

The fundamental idea of divisive algorithms is to identify and remove all the edges that interconnect nodes belonging to different communities by applying an iterative process. At the beginning, the original graph is considered a single community. During each iteration step, a set of edges that meet certain criteria and appear to interconnect different communities, will be removed. The process is repeated until the retrieved network structure converges to a stable state where no additional edge removal can be applied [START_REF] Bisma | Network Community Detection: A Review and Visual Survey[END_REF]. It is worth mentioning that in a few extreme cases the removal of a whole subgraph may be required in terms of community structure validity and efficiency. The most representative divisive algorithm is the one proposed by Girvan and Newman [START_REF] Mark | Michelle Girvan: Finding and Evaluating Community Structure in Networks[END_REF], in which the removal criterion depends on the edge betweenness measure [START_REF] Cukierski | Graph-based features for supervised link prediction[END_REF].

Despite, its algorithmic and conceptual simplicity, this approach is excessively demanding in both calculation and storage resources requirements and hence considered impossible for big data networks. Ample alternatives of Girvan and Newman algorithm have been proposed aiming to substitute the original edge betweenness modularity, such as geodesic or current-flow edge betweenness etc. [START_REF] Fortunato | Community detection in graphs[END_REF] [2], in terms of efficiency increase or complexity reduction. Among the proposed, the most prevalent is the random-walk edge betweenness criterion [START_REF] Liu | MIRACLE: A multiple independent random walks community parallel detection algorithm for big graphs[END_REF] [6], which identifies low density edges that are not part of cycle paths and can be considered as community connection edges.

Agglomerative Algorithms

In contrast to divisive algorithms, the agglomerative are classified as bottom up approaches. By originally considering each node as a separate community, also known as singleton, a number of iterations is applied, in each of which, the distinct communities are merged according to the calculated result of a well-defined similarity function. The ultimate goal is to end up with a graph that compose a unique cluster [START_REF] Fortunato | Community detection in graphs[END_REF].

It is worth pointing out that agglomerative algorithms intrinsically unfold the complete hierarchical community structure of the analysis graph, as new communities are built during the process from previous steps' communities. Even if the generated hierarchies might seem generally artificial, social networks are typical examples of hierarchical structured information since it is natural for people to create groups within their working environment, friends or even families. Among the proposed agglomerative algorithms [START_REF] Satu | Graph clustering[END_REF] [20], the one introduced by the university of Louvain [START_REF] Vincent | Fast unfolding of community hierarchies in large networks[END_REF] is the most widespread and considered as the state-of-the-art in terms of hierarchical community detection. The significant difference of this greedy approach is that the modularity measure used, overtake the rests in terms of simplicity and accuracy. In Louvain's algorithm [START_REF] Vincent | Fast unfolding of community hierarchies in large networks[END_REF], the modularity function ingeniously combines the number of community's intra-connecting edges, the corresponding nodes' degrees and the total number of graph's edges, aiming to identify a final structure with no edges between the densely inter-connected retrieved communities. Extensive research has been done regarding Louvain's alternatives, where the most notable are either trying to combine Louvain's modularity function with other modularity alternatives, as the one presented in [START_REF] Held | Dynamic Clustering in Social Networks using Louvain and Infomap Method[END_REF] that uses Infomap, or to parallelize the structure retrieval procedure [START_REF] Fazlali | Adaptive parallel Louvain community detection on a multicore platform[END_REF] [23].

Transformation Methods

Many researchers of different backgrounds have been engaged to tackle this critical graph clustering problem by trying to transform and project the initial network structure identification problem to a different solution space. The most characteristics are the Spectral Clustering techniques and the adoption of Genetic Algorithms. The Spectral clustering [START_REF] Fortunato | Community detection in graphs[END_REF] techniques are following the transformation methodology where each node is represented by a point in space, whose coordinates are elements of eigenvectors. Consequently, simple clustering techniques, such as K-means clustering, can be applied to come up with the clusters that actually are the original graph's communities [14] [15]. The main drawback of this approach is that the clustering results are seriously poor for graphs originally comprised of many fragmented communities.

Another widely used set of algorithms, in terms of modularity optimization, is the Genetic algorithms [START_REF] Su | Havens: Fuzzy community detection in social networks using a genetic algortihm[END_REF]. Those are particularly repetitive meta-heuristics inspired by the theory of natural evolution. As in the natural evolution process, the original graph is randomly initialized, and each node is assigned to a community [START_REF] Bisma | Network Community Detection: A Review and Visual Survey[END_REF].

Other Approaches

Any of the previously described algorithms try to reveal the underlying community structure for any given network. Although, by only depending on graph's topological structure without taking into account the critical user content information, it is unquestionable that a fundamental piece of social network information is being ignored. As mentioned in [START_REF] Fortunato | Community detection in graphs[END_REF] and [START_REF] Bisma | Network Community Detection: A Review and Visual Survey[END_REF], many approaches have been proposed, such as [START_REF] Wang | High-performance social networking: microblog community detection based on efficient interactive characteristic clustering[END_REF], that try to take leverage the user content information to proceed to a more beneficial graph clustering outcome for user-oriented applications. Additionally, another engaging alternative is the link prediction methodology. This approach's initial intention was the prediction of potential connections between any pair of users that are not currently connected, based on the existing user relationships. However, recent studies [9] [START_REF] Pang | Semi-supervised multi-graph classification using optimal feature selection and extreme learning machine[END_REF] showed that link prediction can also be efficiently used for extensive graph analysis due to its limited computation and resources requirements. Therefore, by slightly modifying the original scope of link prediction, it is obvious than this methodology can equivalently be used to predict whether any pair of directly connected nodes, belong to the same community or not, which is the basic idea of this paper.

Other equally important algorithms and methodologies such as Simulated Annealing [START_REF] Fortunato | Community detection in graphs[END_REF], Information Diffusion Graph Clustering [START_REF] Bisma | Network Community Detection: A Review and Visual Survey[END_REF], External Optimization [START_REF] Fortunato | Community detection in graphs[END_REF] and Generative Models [START_REF] Satu | Graph clustering[END_REF] are not further discussed since they are out of the scope of this paper.

Proposed Methodology

Graph Statistical Analysis & Feature Enrichment

This is the initial processing step where complementary to graph's statistical analysis, a predefined set of features is assigned to each graph's node and edge accordingly.

The statistical analysis regarding the original graph's topology structure is mandatory in order to ensure that the consequently extracted subgraph would be representative to the original. As a result, the average node degree, which is the average number of connections per node, and the graph's degree distribution, which is the probability distribution of node degrees over the whole network, need to be calculated. Regarding the feature enrichment process, a predefined set of features, separately per node and edge, is calculated and assigned accordingly. It is worth pointing out that the amount of information implied only by the edge and its adjacent nodes, might not be sufficient for proper prediction. Undeniably, each connection's properties are not only affected by their direct adjacent nodes, but also substantially influenced by their surrounding network structure. Hence, the current level predictors should be enriched with the values of all of the corresponding features up to a predefined depth k. Empirically, the features that correspond to depths greater than 5, do not add any essential information since they tend to converge to graph's average values. On the contrary, the experimentation process, which is comprehensively analyzed in the following subsection, has practically proven that the community prediction is substantially benefited with feature enrichment up to the 3rd connection level.

The only node-oriented feature calculated is the number of k-depth neighbors. Since this algorithm's ultimate scope is the edge-oriented prediction, all calculated feature's values for each engaging node will be included as impeding edge's additional information.

The edge-oriented prediction features are:

• The loose similarity: regarding an imminent pair of nodes and each of their corresponding sets of distinct nodes up to depth k, this is the fraction of common over the union set's total number of nodes. It should be underlined, that both sets include nodes having depth less or equal to k, since a common node might be at a different depth for each impeding node. • The dissimilarity: regarding an imminent pair of nodes and each of their corresponding sets of distinct nodes up to depth k, this is the fraction of uncommon over the union set's total number of nodes. • The edge balance: regarding an imminent pair of nodes and each of their corresponding sets of distinct nodes up to depth k, this is the fraction of the absolute difference between the peers adjacent over the union set's total number of nodes. This feature values ranges from 0 to 1, with values closer to 0 indicating a balanced edge that have merely equal k-depth vertices on both sides. • The edge information: considering that the corresponding element on a graph's kth power adjacency matrix indicates the number of distinct k-length paths between the impeding vertices [START_REF] Kranda | The Square of Adjacency Matrices[END_REF], this feature reveals the pair's interconnection strength.

Representative Subgraph Extraction

To properly train a community prediction model, the knowledge of Louvain algorithm's behavior regarding the original graph is considered radical. Nevertheless, considering the size of real-world social graphs, it is obvious that even the execution of Louvain's distributed implementation [START_REF]Sotera Distributed Graph Analytics (DGA): Sotera Defence Solution[END_REF] can be considered prohibitive, since it should still need to repetitively check an immense number of candidates per iteration in order to merge the ones with the highest probability of forming a community. Thus, the extraction of a subgraph that retain the original graph's elementary topology properties is deemed fundamental and that is indeed the main purpose of this processing step. After thorough investigation, the Tiny Sample Extractor algorithm [START_REF] Sethu | A new algorithm for extracting a small representative subgraph from a very large graph[END_REF] has been selected for its well efficiency and high scalability. Even if the extraction of a representa-tive subgraph yet remains controversial, since the real networks exhibit the degree distribution that follow the power law, the extracted subgraph's degree distribution remains remarkably similar to the original graph's, as clearly shown in Figure 1.

Fig. 1. The exponent degree distribution on log-log scale for Email-Eu-Core [START_REF] Klimmt | Introducing the Enron corpus[END_REF] dataset.

Having already calculated the original graph's degree distribution during the previous graph analysis processing step, the Tiny Sample Extractor [START_REF] Sethu | A new algorithm for extracting a small representative subgraph from a very large graph[END_REF] algorithm initially tries to estimate the original graph's exponent degree by the slope of the log-log Complementary Cumulative Distribution Function (CCDF). Bear in mind, that the CCDF function for a given degree value d, returns the fraction of nodes that have degree greater than d. Therefore, starting from a random node, 3 biased random walks are applied in turn. In the first couple of walks, the minimum and maximum exponent degree values are respectively defined. In the last one, the resulting subgraph is extracted. The bias included is used in terms of compensation due to the skewed distribution of nodes visited in the random walks performed.

In the extreme case, where the original graph is that big, in such extent that is not even traversable, then both the previously described statistical analysis and the representative subgraph extraction step can be applied only to a smaller part of the original graph that would is considered as the graph's "known" part.

Community Prediction Model Training & Application

The most suitable machine learning model for prediction, in terms of big data efficiency and interpretability, is undoubtedly the logistic regression model. Projecting its application to community detection, the eventual scope of this model would be to predict whether a given edge connects nodes of the same community or not, depending on the aforementioned edge-oriented graph topology features. Despite the intrinsic accuracy drawbacks of linear models, it is frequently proven that on real-worlds problems, they seem to be surprisingly competitive to non-linear thanks to their low variance.

To appropriately train a logistic regression model, the classification response needs to be predefined at the training dataset. Among the copious proposed methods, the Louvain's one [START_REF] Vincent | Fast unfolding of community hierarchies in large networks[END_REF] is broadly considered [6] [20] as the state-of-the-art in disjoint community detection, while its performance generally serves as the golden baseline. So, using the previously extracted subgraph, the distributed version of Louvain algorithm [START_REF]Sotera Distributed Graph Analytics (DGA): Sotera Defence Solution[END_REF] is applied to classify all subgraph's edges as "community" or "non-community".

As it is broadly known, since Louvain's returned community structure is excessively affected not only by the traversal fashion (BFS, DFS) but also from the starting candidate that the execution starts, it would be prudent not only to re-execute it starting from different nodes but also to use different traversing methodologies. Even if this repetition might sound extremely demanding, the size of the extracted subgraph significantly restricts the required calculations.

The large number of descriptive topology features supposed to improve the model prediction accuracy. However, it is very usual for some or many input features to either not be truly associated with the response or to be highly correlated with each other. The first case, also known as the "curse of dimensionality", refers to the situation where irrelevant features are included during training, while the second, also known as the "collinearity effect" occurs when two or more predictors are correlated to one another. Both effects lead to unnecessary complexity introduction and interpretability reduction. Consequently, during the training process the model should automatically proceed to feature selection by applying the L1 regularization penalty that excludes the redundant features and practically performs feature selection. Consequently, using the Spark MLlib's [START_REF] Meng | MLlib: Machine Learning in Apache Spark[END_REF] distributed libraries, a L1 regularized distributed logistic regression classifier is trained, using the previously extracted subgraph as training dataset and the enrichment features described on the first processing step as predictors. This model prediction is used to identify the original graph's underlying community structure.

Experiments

To evaluate the performance of the proposed methodology, its predicted structure is compared against the communities returned from the distributed Louvain implementation [START_REF]Sotera Distributed Graph Analytics (DGA): Sotera Defence Solution[END_REF]. For this experimentation procedure, the following widely used and thoroughly analyzed social graphs are evaluated. It is noteworthy that the first two [START_REF]Zachary karate club network dataset --KONECT[END_REF] [25] datasets considered reference datasets, since they have historically attracted the scientific interest and have exhaustively been analyzed. All the experiments were executed in an eight node Spark 2.2.0 cluster, with 4 GBs RAM and 1 virtual core per each virtual machine. Even if the processing performance is out of the scope of this comparison, it is worth mentioning that the proposed algorithm's processing time and memory requirements were constantly less, at least 25% and 55% respectively, comparing to the distributed Louvain implementation [START_REF]Sotera Distributed Graph Analytics (DGA): Sotera Defence Solution[END_REF]. The performance improvement will certainly be substantially higher in bigger social graphs, since the proposed methodology's complexity is linear and not of a higher polynomial degree as in Louvain's case.

In terms of completeness, all the elementary classification performance metrics have been calculated. Specifically:

• The accuracy, which is the number of edges correctly classified as either "community" or "non-community" over all the predictions made. • The precision, which is the number of edges correctly classified as "non-community" over the total number of "non-community" predictions. • The recall/sensitivity, which is the fraction of edges correctly classified as "noncommunity" over the total number of truly "non-community" edges. • The specificity, which is the fraction of edges correctly classified as "community" over the total number of truly "community" edges. The community detection results regarding the aforementioned social graphs are presented in the following table, Table 2. As it is shown in Table 2 the proposed prediction model shows excellent overall prediction performance with splendid statistics on precision and specificity. Specifically, the trained prediction model shows magnificently rational general accuracy that reach the 80.6% in the average case. In other words, the proposed algorithm remarkably identifies each edge's class in general. Focusing on the outstanding specificity and precision statistics, it is more than obvious that the proposed implementation impressively identifies the "community" edges. This way, the graph's consistency is preserved by retaining social graph's true connections and avoiding graph's fragmentation. However, the mediocre recall statistics shows that the trained model misclassifies several "non-community" edges as "community" ones. This will doubtlessly lead to a less fragmented community structure comparing to the one returned from original Louvain's algorithm. Actually, this can be graphically confirmed by the following figure, Figure 2 where the Email-Eu-Core [START_REF] Sethu | A new algorithm for extracting a small representative subgraph from a very large graph[END_REF] graph's returned community structure is presented in both cases, having the red colored edges stand for the "non-community" edges and the blue ones for the "community". As presented, the returned community structure is eminently similar in both cases. The "non-community" edges are apparently less in the proposed community prediction algorithm situation. However, the use of more complex machine learning models, such as non-linear prediction ones, is supposed to catapult the recall statistics, since linear prediction models inherently focus on interpretability and not on prediction's accuracy.

Conclusions

In this manuscript, a novel distributed community prediction methodology has been introduced based on Louvain's [START_REF] Vincent | Fast unfolding of community hierarchies in large networks[END_REF] community detection algorithm. Unquestionably, the experimentation process confirmed the remarkable performance in identifying the subjacent community structure of social graphs comparing to Louvain's original algorithm. Despite the encouraging results, it is obvious that the methodology's community prediction can be substantially improved by: • Applying a graph cleansing processing step, where the outliers' removal will be applied.

• Combing the collinear variables into a single predictor to tackle the multi-collinearity effect. • Applying less interpretable but more accurate models, e.g. non-linear prediction models. But all the above are left for future work.

Fig. 2 .

 2 Fig. 2. Email-Eu-Core [17] Dataset Community Detection comparison

Table 1

 1

		. Evaluated Datasets
	Dataset	# of Nodes	# of Edges	Edges / Nodes
	Zachary karate club [24]	34	78	2.29
	Dolphins [25]	62	159	2.56

Table 2 .

 2 Classification Performance Metrics

	Dataset	Accuracy	Precision	Recall	Specificity
	Zachary karate club [24]	76.5%	100%	50%	100%
	Dolphins [25]	88.2%	91.7%	78.6%	95%
	Hamster friendships [26]	69.7%	72.3%	54.5%	82.5%
	email-Eu-core [27]	78.1%	81.8%	80.2%	75.2%
	Bitcoin Alpha [28]	89%	92.6%	14%	99.8%
	Enron email [29]	82%	73.2%	27.7%	97.2%

Acknowledgment

Christos Makris is co-financed by the European Union (European Social Fund) and Greek national funds through the Operational Program "Research and Innovation Strategies for Smart Specialization -RIS3" of "Partnership Agreement (PA) 2014-2020".


Siyu Yu
email: yusiyu@stu.xjtu.edu.cn

Nanning Zheng
email: nnzheng@mail.xjtu.edu.cn

Hao Wu

Ming Du

Badong Chen
email: chenbd@mail.xjtu.edu.cn

Exploring Brain Effective Connectivity in Visual Perception Using a Hierarchical Correlation Network

Keywords: Brain-inspired computing, visual perception, functional magnetic resonance imaging (fMRI), Hierarchical correlation network (HcorrNet), connection

Brain-inspired computing is a research hotspot in artificial intelligence (AI). One of the key problems in this field is how to find the bridge between brain connectivity and data correlation in a connection-to-cognition model. Functional magnetic resonance imaging (fMRI) signals provide rich information about brain activities. Existing modeling approaches with fMRI focus on the strength information, but neglect structural information. In a previous work, we proposed a monolayer correlation network (CorrNet) to model the structural connectivity. In this paper, we extend the monolayer CorrNet to a hierarchical correlation network (HcorrNet) by analysing visual stimuli of natural images and fMRI singals in the entire visual cortex, that is, V1, V2 V3, V4, fusiform face area (FFA), the lateral occipital complex (LOC) and parahippocampal place area (PPA). Through the HcorrNet, the efficient connectivity of the brain can be inferred layer by layer. Then, the stimulus-sensitive activity mode of voxels can be extracted, and the forward encoding process of visual perception can be modeled. Both of them can guide the decoding peocess of fMRI signals, including classification and image reconstruction. In the experiments, we improved a dynamic evolving spike neuron network (SNN) as the classifier, and used Generative Adversarial Networks(GANs) to reconstruct image.

Introduction

A better understanding of the brain can inspire researchers for efficient methods in brain-inspired computing, which can help to develop more robust artificial intelligence (AI) technologies. [START_REF] Zheng | Hybridaugmented intelligence: collaboration and cognition[END_REF]. In recent decades, brain-inspired computing with functional magnetic resonance imaging (fMRI) signals have driven plenty of studies [START_REF] Hausfeld | Multiclass fmri data decoding and visualization using supervised self-organizing maps[END_REF][START_REF] Naselaris | A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes[END_REF].

One key problem is how to analyze the correlations in collected data for modeling connection-to-cognition process. Much of the literature are devoted to modeling the correlations between neural activities in visual perception and revealing neural connections in brain [START_REF] Norman | Beyond mind-reading: multi-voxel pattern analysis of fmri data[END_REF][START_REF] Kay | Identifying natural images from human brain activity[END_REF][START_REF] Horikawa | Generic decoding of seen and imagined objects using hierarchical visual features[END_REF]. In early studies, voxel receptive-field models have been used to analyze the correlations by locating a small visual stimulus that elicits activities spatially spread over voxels. Some researchers combined linear regression and Bayesian inference algorithms with correlation models [START_REF] Yamashita | Sparse estimation automatically selects voxels relevant for the decoding of fmri activity patterns[END_REF][START_REF] Penny | Statistical parametric mapping: the analysis of functional brain images[END_REF]. In recent years, deep neural network (DNN), VAEs [START_REF] Kingma | Auto-encoding variational bayes[END_REF] and DCCAE [START_REF] Wang | On deep multi-view representation learning[END_REF] have been used to study this problem. As is well known, neural connections can be divided into three types of connections between brain nodes: functional connectivity, structural connectivity, and effective connectivity [START_REF] Park | Structural and functional brain networks: from connections to cognition[END_REF][START_REF] Yu | A novel brain decoding method: a correlation network framework for revealing brain connections[END_REF]. However, most existing correlation models [START_REF] Fujiwara | Modular encoding and decoding models derived from bayesian canonical correlation analysis[END_REF][START_REF] Kuang | Discrimination of adhd based on fmri data with deep belief network[END_REF] have left out the structural information of voxel, which may provide information about the intracortical or intrinsic connections, i.e., the structural connectivity. To reveal the working mechanism of the human brain (connection-to-cognition process), in a previous work, we proposed a monolayer correlation network (CorrNet) by considering the three correlations to simulate the rich effective connectivity of the early visual cortex and establish mapping from visual perceptual tasks to voxels [START_REF] Yu | A novel brain decoding method: a correlation network framework for revealing brain connections[END_REF].

Recenlty, some DNNs [START_REF] Shen | Deep image reconstruction from human brain activity[END_REF] and hierarchical structure [START_REF] Eliasmith | A large-scale model of the functioning brain[END_REF] are utilized to build braininspired models. As for image reconstruction, some reseachers use fMRI signals as the input of Gans [START_REF] Shen | End-to-end deep image reconstruction from human brain activity[END_REF]. Inspired by these works, in this paper, we propose a hierarchical correlation network (HcorrNet) by extending our monolayer CorrNet [START_REF] Yu | A novel brain decoding method: a correlation network framework for revealing brain connections[END_REF] to deep framework case. The study rests on analyzing hierarchical correlations to explore effective connectivity in the entire visual cortex (VC), including V1, V2, V3, V4, fusiform face area (FFA), lateral occipital complex (LOC), parahippocampal place area (PPA). In the HcorrNet, the correlations of collected data are obtained, including perception-voxel corresponding correlation and voxel-voxel correlation. The voxel-voxel correlation can be modeled as voxel correlation between voxels in the same visual cortex and the hierarchical voxel correlation between voxels of two layers. Then, the correlations are utilized to model the stimulus-sensitivity activation patterns of brain and the layer-tolayer forward propagation process, that is, the forward encoding information flow, both of which help to explore the effective connectivity in visual perception. Then, based on the inferred effective connectivity, we location the most activate voxels for the decoding process, including classification and image reconstruction. In our experiments, a improved dynamic evolving spike neuron network (SNN) [START_REF] Gütig | The tempotron: a neuron that learns spike timing-based decisions[END_REF][START_REF] Hu | How the brain formulates memory: A spatio-temporal model research frontier[END_REF] and Wasserstein GAN (WGAN) [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] are used to classify and reconstruct image, respectively. The experimental dataset is the public fMRI dataset from ATR Computational Neuroscience Laboratories of Kyoto University [START_REF] Horikawa | Generic decoding of seen and imagined objects using hierarchical visual features[END_REF].

The rest of the paper is organized as follows. Section 2 describes the monolayer CorrNet. Section 3 proposes the HcorrNet framework. Section 4 describes the decoding process, including classification adn reconstruction. In section 5, experiments are conducted to demonstrate the performance of our method. Finally, conclusion is given in section 6.

Monolayer Correlation Network

Our previous work rested on how the visual cortex organizes addresses different visual stimuli based on function-structure relationships in the brain. Three kinds of correlations were modeled, including the correlation among brain activities, the correlation between brain activities and visual stimuli, and the correlation among visual stimuli. We transformed the correlation degree into correlation probability, and defined the following novel concepts:

Voxel correlation probability measures the voxel correlations, whose element is the probability that any two voxels are of functional and structural correlation.

Strength correlation probability measures the strength correlation, which means the similarity or opposite reflection to the same stimuli. Its element presents the probability of strength correlation.

Topological correlation probability measures the topological correlation, which means any two voxels are in the same topological space. Its element presents the probability that any two voxels are of topological correlation.

Corresponding correlation is viewed as a significance measure of the corresponding correlation between one voxel and one pixel, which is not of simple linear relationship, and is different from the traditional correlation measure, such as Pearsons Correlation.

Corresponding correlation probability is the probability that one voxel of is in the inverse receptive field of one pixel. Its values determines corresponding correlation.

In the monolayer CorrNet, above correlations refers to the functional connectivity, the structural connectivity and the effective connectivity. So, we introduced the topological correlation to measure the structural connectivity. Notably, Delaunay triangulations are utilized to reveal the topological correlation, because Delaunay triangulations can reveal present superficial connectivity between voxels, which is consistent with the inner structure of the cerebral cortex [START_REF] Zeeman | Topology of the brain[END_REF]. The functional connectivity was measured by the Strength correlation, which is computed in a full covariance matrix. Then, Bayes rules were used to combine topological correlation with strength correlation between voxels to obtain voxel correlation. Later, we computed the corresponding correlation between voxels and pixels in a probabilistic correlation graph. In the graph, the corresponding correlation probability was initialized as strength correlation probability between voxel and pixel. Next, considering the voxel correlation, the graph was updated, iteratively, by computing the joint probability of voxel correlation and strength correlation between voxel and pixel. Then, a threshold parameter was used to determine the degree of the corresponding correlation. When the corresponding correlation probability of certain voxel-pixel pair is equal to or greater than the threshold parameter, the voxel and pixel are of significant corresponding correlation, that is, the voxel is in the inverse receptive field of the pixel. Finally, the inverse receptive field of the pixel in brain was located, which revealed the effective connectivity in visual cognition tasks.

Hierarchical Correlation Network

As Fig. 1 shows, to explore the entire visual cortex, this paper focuses on how to model the correlations between stimuli and the voxels int the entire visual cortex, including V1, V2, V3, V4, FFA, LOC, PPA. Considering that the cognition information of visual stimuli is forward encoded from lower visual cortex to higher visual cortex, we propose a deep framework with hierarchical correlation network (HcorrNet) to address this problem. Thus, the original problem, exploring the effective connectivity in visual perception, is translated into three sub-problems: How to obtain the correlations inferred by the collected data, including voxel correlation between voxels in the same visual cortex and the voxel correlation between voxels in two layers (such as V1 and V2) and perception-voxel corresponding correlation (pixel-voxel corresponding correlation). We call the former inner-cortex voxel correlation, and compute the probability that two voxels are of inner-cortex voxel correlation, denoted as P rob vinn . Similarly, the latter is called inter-cortex voxel correlation, P rob vint is defined as the probability that two voxels are of inter-cortex voxel correlation;

How to use the correlations to explore the stimulus-sensitivity activation patterns; How to represent the correlations in the forward encoding process.

Voxel Correlation in the HcorrNet

The inner-cortex voxel correlation in each layer, denoted as Cv lc inn can be obtained by our previous work of CorrNet [START_REF] Yu | A novel brain decoding method: a correlation network framework for revealing brain connections[END_REF], easily, which exactly analyses the voxel correlation in single layer.

As for the inter-cortex voxel correlation, the computation procedure is different from that of the inner-cortex voxel correlation. In this paper, lc, derived from V1, V2, V3, V4, FFA, LOC, PPA, is denoted a certain visual cortex. vn lc denotes the number of voxels in the lc th layer. V lc i ∈ R N , i = 1, ..., vn lc stands for the strength value vector of the i th voxel in the lc th visual cortex, N is the number of trials. V num is the number of all voxels in the entire visual cortex. v lc i,m , i = 1, ..., vn lc , m = 1, ..., N represents the m th element of

V lc i . P rob vint (V lc i , V lc+1 j
) is denoted as the inter-cortex voxel correlation between the i th voxel in the lc th layer and the j th voxel in the lc + 1 th layer. The strength correlation probability of any two voxels from adjacent visual cortex is denoted as P rob vstr , given by:

P rob vstr (V lc i , V lc+1 j )= N m=1 (v lc i,m v lc+1 j,m )- N m=1 v lc i,m N m=1 v lc+1 j,m N N m=1 v lc i,m 2 - N 1 2 v lc i,m N N m=1 v lc+1 j,m 2 - N 1 2 v lc+1 j,m N (1)
Then, P rob vint of two voxels V lc i , i = 1, ..., vn lc and V lc+1 j , j = 1, ..., vn lc+1 is given by Algorithm .1.

A voxel correlation matrix Cv lc inter ∈ R vn lc ×vn lc+1 is introduced to simplify P rob vint .

Cv lc inter (V lc i , V lc+1 j
) defines the correlation between voxel V lc i and voxel V lc+1 j , whose computation process is detailed in [START_REF] Yu | A novel brain decoding method: a correlation network framework for revealing brain connections[END_REF].

When P rob vint (V lc i , V lc+1 j ) >= thre vint , Cv lc inter (V lc i , V lc+1 j ) = 1, otherwise, Cv lc inter (V lc i , V lc+1 j ) = 0
, where thre vint is a free parameter, used to determine the degree of correlation and set to 0.5 in the present study.

Corresponding Correlation in the HcorrNet

Following our previous work [START_REF] Yu | A novel brain decoding method: a correlation network framework for revealing brain connections[END_REF], we compute the corresponding correlation between visual perception and voxels layer by layer. More precisely, we compute the corresponding correlation by using the monolayer algorithm, independently and repeatedly. For each layer, a binary matrix Corr lc ∈ R P num×vn lc , lc ∈ {1, ..., 6} is denoted as the corresponding correlation matrix of the lc th , where P num is the number of pixels. Thus, Corr lc (p k , V lc i ) = 1 means the i th voxel of the lc th visual cortex is in the inverse receptive field of the k th pixel, where p k is the k th pixel of images.

Stimulus-sensitivity Activation Patterns

As is known, certain fixed activation patterns of Stimulus-sensitivity character can be inferred from human brain when the stimuli are of the same category, and can be viewed as stimulus-sensitivity cognition unit [START_REF] Wen | Neural encoding and decoding with deep learning for dynamic natural vision[END_REF]. In this paper, to explore how the different stimuli affect the brain connectivity, we compute the joint probability distribution of all voxel correlation probability matrixes with category labels, denoted as Algorithm 1 Generating P rob vint .

Require:

P robvinn of all voxels in the lc th , P robvstr of the lc and lc + 1 layers, a threshold, V lc i , i = 1, ..., vn lc and V lc+1 j , j = 1, ..., vn lc+1 , strength correlation threshold εvstr, inter-cortex voxel correlation threshold εvinn. Ensure:

P robvint(V lc i , V lc+1 j ) 1: Initial P robvint(V lc i , V lc+1 j ) = P robvstr(V lc i , V lc+1 j ) 2: for i = 1 to vn lc do 3: temp = 0; Li = φ; 4:
for j = 1 to vn lc+1 do 5:

temp = max(temp, P robvint(V lc i , V lc+1 j )), 6: if P robvstr(V lc i , V lc+1 j
) ≥ εvstr then 7:

kt = M AXN U M ; 8:

for j * = 1 to vn lc+1 do 9: ) ∈ {0, 1} can be viewed as the common effective connectivity with category guidance, measuring whether the i th voxel of the lc th visual cortex and the j th voxel of the (lc + 1)

if P robvinn(V lc+1 j , V lc+1 j * ) ≥ εvinn then 10: Update Li = Li ∪ {j * }; kt = min(kt, d(V lc+1 j , V lc+1 j * )); for all m ∈ Li, calculate: pinn = kt/d(V lc+1 j , V lc+1 m )); P robvint(V lc i , V lc+1 m ) = pinn * P robvint(V lc i , V lc+1 j ); P robvint(V lc i , V lc+1 m ) = max(temp, P robvint(V lc i , V lc+1
th visual cortex are of stimulus-sensitivity voxel correlation or not. Similarly, the common effective connectivity relying on the pixel-voxel corresponding correlation, denoted as Cpv lc ss can be obtain by computing the joint probability distribution of all corresponding correlation matrixes Corr lc derived from the stimuli within the same category with Hadamard products. is a binary matrix Cpv lc ss (p k , V lc i ) = 1 measures whether the i th voxel of the lc th visual cortex and the k th pixel are of stimulus-sensitivity corresponding correlation or not. For each pixel p k , a set Cr lc k = {i | Cpv lc ss (p k , V lc i ) = 1} is defined to present the set of voxels in the lc th visual cor-Algorithm 2 obtain Cv lc ss .

Require: Cv lc inter , Label ∈ R N is the category label, classnum = M AX(Label), v lc i,m , i = 1, ..., vn lc , m = 1, ..., N , v lc+1 j,m , j = 1, ..., vn lc+1 , m = 1, ..., N . Ensure:

Cv lc ss ∈ R vn lc ×vn lc+1 1: Initial Cv lc ss (V lc i , V lc+1 for i = 1 to vn lc do 5:

Vlc i = V lc i (list) 6:
for j = 1 to vn lc+1 do 7:

Vlc+1 j = V lc+1 j (list); temp Cv lc ss (i, j) = Cv lc inter (V lc i , V lc+1 j ); 8:
end for 9:

end for Cv lc ss . × temp Cv lc ss ; 10: end for tex which have stimulus-sensitivity corresponding correlation with p k . The voxels in Cr lc k can be viewed as active nodes in terms of stimulus-sensitivity activation patterns.

Forward Encoding Process Representation in the HcorrNet

It is believed that, the higher visual cortex receives the information from the lower visual cortex. In this paper, we try to model the information flow process. Firstly, we build a directed probability graph to figure out which voxels in the higher visual cortex will be passed by the information from certain voxel in the lower visual cortex. Notably, our method with bayes reference is of data-diven analysis, which is totally different from the anatomical methods. Those methods mainly tend to use biomarkers to trace the information flow. In the garph, P rob f w is denoted as information-reachable probability, presenting the binary probability that any two nodes (voxels) are information-reachable. P rob f w (V lc1 i , V lc2 j ), lc1 < lc2 presents whether the information can flow from the voxel V lc1 i in the lower layer to V lc2 j in the higher layer or not. P rob f w is given by:

P rob f w (V lc1 i , V lc2 j ) = lc2-1 k=lc1 Cv lc ss (V k i , V k+1 j ) (2)
Next, we complete the whole forward encoding process by viewing the visual image as the 0 th layer and computing P rob f w (p k , V lc m ), k = 1, ..., P num, m = 1, ..., V num, lc is the layer tag of the m th voxel within all voxels. When lc = 0,P rob f w (p k , V lc m ) = Cpv lc ss (p k , V lc+1 m ), otherwise:

P rob f w (p k , V lc m )= m * ≤vn lc max m * =1 (P rob f w (p k , V lc-1 m * ) • P rob f w (V lc-1 m * , V lc m )) (3)
When P rob f w (p k , V lc m ) = 1, the k th pixel and the m th voxel with layer tag lc are information-reachable pair. For each pixel p k , a set Fr lc k = {m | P rob f w (p k , V lc m ) = 1} is defined to represent the set of voxels in the lc th visual cortex which are informationreachable to p k . The voxels in Fr lc k can be viewed as effective nodes in the forward encoding process. The more voxels Fr lc k has, the more voxels the effective connectivity in terms of the corresponding pixel p k is related to. In addition, a chain from certain pixel to certain voxel in the HVC can be revealed, which can be viewed as the whole the information flow from visual perception to brain response. Notably, to simplify the computation, the forward encoding process is computed in chain V1 → V2 → V34 → HVC.

Based on the pixel-wise inverse receptive field, we compute the pixel-voxel correlation pairs by combining the forward encoding process representation and stimulussensitivity activation patterns, denoted as bin k for k = 1, ..., P num. bin k presents the set of voxels that are both within the information flow of pixel p k , and are of similar stimulus-sensitivity activation pattern. bin k is:

bin k = {i|Cv hvc ss (p k , V i ) = 1&P rob f w (p k , V i ) = 1} (4)
Then, to location the voxels with the most high active stimulus-response pattern, we use sliding windows in the image coordinate to union the intersectional voxels in of bin k within each window, denoted Voxel act . Voxel act contains all voxels with the most high active stimulus-response pattern, and can be used in decoding process, including classification and reconstruction.

Decoding Process

In the decoding process, only voxels in Voxel act are used. As is known, the learned connections of SNNs [START_REF] Hu | How the brain formulates memory: A spatio-temporal model research frontier[END_REF] represent dynamic spatiotemporal relationships, making SNNs more biomimetic than artificial neural networks (ANNs) in both information transmission and structure. In our previous work, SNNs have been used to classify MNIST Dataset and fMRI signals of simple images, yielding satisfying results. Therefore, in this paper, a similar SNN structure is also used as the classifier. Fig. 2 shows the network structure of the SNN. In the training process, the input of the SNN is a N × M matrix with N training samples(N training trials) and M voxels response attributes (M is the number of voxels in Voxel act ), the output is the class label of corresponding stimulus (50 in this paper). The detail is in [START_REF] Yu | A novel brain decoding method: a correlation network framework for revealing brain connections[END_REF][START_REF] Ma | A novel spiking neural network of receptive field encoding with groups of neurons decision[END_REF].

To obtain reconstruction images, a improved WGAN [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] with gradient penalty is trained to generate 32 × 32 × 3 image. The input of generator is fixed voxel response value scalars of all M voxels in Voxel act corresponding to the related stimulus, instead of random noise. The input images (32×32×3) is transformed from the ordinary images (RGB, 800 × 800 × 3).

Experiments

The experimental data comes from the public fMRI dataset from ATR Computational Neuroscience Laboratories of Kyoto University, which contains training set and test set. The training set includes 1200 nature images of 150 categories (8 images from each category) and the corresponding fMRI signals. In the process with HCorrNet, we only used the training set to obtain the voxel tag Voxel act . Firstly, the nature images (RGB, 800 × 800 × 3) were transformed into grayscale images and resized into 224 × 224 with normalization process. Moreover, V3 is merged into V4 as V34 layer. In the decoding process, we use the test set which includes 50 nature images of new 50 categories (never intersect with the category in the training set). Each image in the test set is showed 35 times to subjects, repeatedly. The first 80% of the test set is used as the training set for classification and reconstruction, the rest is used as the test set in the decoding process.

As detailed before, Cr lc k = {i | Corr ss (p k , V lc i ) = 1} is defined to represent the tag set of voxels in the lc th layer which have stimulus-sensitivity corresponding correlation with p k . Fr lc k = {m | P rob f w (p k , V lc m ) = 1} is defined to indicate the tag set of voxels in the lc th layer which are information-reachable to p k . When lc presents the HVC region, the forward encoding is accomplished. So, the reservation rate of voxels in the Voxel act is showed in Tabel 1(Note: there is overlap between the layers.): voxels in Fr lc k into the coordinates of p k in the image coordinate frame in chain V1 → V2 → V34 → HVC, as shown in Fig. 3. The first four images is the quantitative distribution of V1, V2, V34 and HVC from Fr lc k , and the last one is the quantitative distribution of the union set of voxels of Fr lc k . The more red the pixel patch, the more voxels p k is information-reachable to. The more red the k th pixel patch, the more voxels Fr lc k contains. In the last heat map, the middle region contains the most voxels, which conforms to the attention mechanism in human perception. Additionally, more higher the layer is, more conspicuous the tendency that most voxels concentrate on the middle region is. Therefore, our HCorrNet framework can reveal the attention mechanism in the forward encoding process well. The classification result of our SNN is 68.43%. We failed to find other similar research of fMRI classification by using nature images as stimuli. Thus, there is no comparison. However, it is hard to process fMRI signals because of its severe noise and high dimension. Thus, 68.43% for 50 categories is accredited and maybe yield the state of art performance. In addition, the reconstruction result is shown in Fig. 4. This reconstruction result can be nearly recognizable and significantly different, which demonstrates the HCorrNet reserves important voxels, and reveals the efficient connectivity and data correlations, effectively.

Through HCorrNet, less than 4% voxels are selected (see Tabel 1), which confirms the excellent denoising performance of our methods. The decoding result demonstrates the importance of HCorrNet in the entire visual cortex as it not only refines the highdimension fMRI signals for decoding task, but also enhances the data correlations. Additionally, the information flow in the forward encoding process has been revealed both in the algorithm level and in the visualization level (see Fig. 3).

Conclusion

In this paper, authors propose a hierarchical correlation network (HcorrNet) to explore the effective connectivity in visual perception by modeling the correlations between perceptual images and voxels from fMRI signals in the entire visual cortex. Our Hcor-rNet addresses three key problems: how to obtain the correlations, how to explore the stimulus-sensitivity activation patterns on the basis of correlations, and how to model the forward encoding process. Then, by combining the stimulus-sensitive activity patterns and the forwarding process, the most activate voxels in the inferred effective connectivity can be located to guide the decoding process, including classification and image reconstruction. The experimental results of classification and image reconstruction demonstrate that the HcorrNet refines the fMRI signals and enhances the data correlations. Our work can contribute to revealing the working mechanism of the human brain and establishing mapping from perceptual tasks to voxels. However, reliable contrast experiments are desired, and experiments on other fMRI datasets are required to further improve and generalize the HCorrNet framework for practical applications in the future.

Fig. 1 .

 1 Fig. 1. The framework of our methods.

j) = 1 , 2 :

 12 temp Cv lc ss = Cv lc ss . for k = 1 to classnum do 3: list = {tag | Label(tag) = k, tag = 1, ..., N }; 4:

Fig. 2 .

 2 Fig. 2. The structure of our SNN.

Fig. 3 .

 3 Fig. 3. Quantitative distribution of voxels.

Fig. 4 .

 4 Fig. 4. Reconstructions result (Subject 1) of some images in the test set. The perceptual images are shown in the first row. The second is the processed images which are the input images of the WGAN, the rest four rows are the RGB reconstruction images of different test trials.

Table 1 .

 1 The reservation rate of voxels in the Voxelact.

	Visual Cortex	V1 V2 V3 V4 FFA PPA LOC LVC HVC Entire
	All Voxel Number	1004 1018 759 740 540 568 356 2281 2049 4466
	Reserved Voxel Number 69 132 18 2	0	3	14 136 27	163
	Reservation(%)	6.87 12.97 2.37 0.27 0.00 0.51 3.93 5.96 1.32 3.65
	To visualize the information flow, five heat maps are drawn to reveal the quantitative
	distribution of voxels in the Fr lc k , k = 1, 2, ..., 224 × 224 by mapping the number of

This work was supported by the National Natural Science Foundation of China (NO. 61773312, L1522023 and No. 91648208) and the 973 Program (No. 2015CB351703).


Neha Bansal
email: nehabansal33@gmail.com

Arun Sharma
email: arunsharma@igdtuw.ac.in

R K Singh
email: rksingh@igdtuw.ac.in

A Review on the Application of Deep Learning in Legal Domain

Keywords: Deep Learning, Legal Text analytics, Classification, Prediction systems

The Amount of legal information that is being produced on a daily basis in the law courts is increasing enormously and nowadays this information is available in electronic form also. The application of various machine learning and deep learning methods for processing of legal documents has been receiving considerate attention over the last few years. Legal document classification, translation, summarization, contract review, case prediction and information retrieval are some of the tasks that have received concentrated efforts from the research community. In this survey, we have performed a comprehensive study of various deep learning methods applied in the legal domain and classified various legal tasks into three broad categories, viz. legal data search, legal text analytics and legal intelligent interfaces. The proposed study suggests that deep learning models like CNNs, RNNs, LSTM and GRU, and multi-task deep learning models are being used actively to solve wide variety of legal tasks and are giving state-of-the-art performance.

Introduction

The continued application of computational intelligence in legal domain has been going on for last few decades. With the increased availability of legal text in digital form, the focus on developing intelligent models and applications have received concentrated rationale from the research community. A wide variety of issues, including summarization, reasoning, classification, translation, text analytics, and others have been applied to a range of legal domain problems. The usage of computer-based intelligent support has many-fold benefits for the legal professional community. These benefits include reducing the laborious human task involved in searching and retrieval of relevant material, reducing the legal costs via automation; resolving or settling issues without the involvement of courts or with less time and effort; negotiation of the law for legal professionals and also the common users; and making decisions based on prediction systems which may be considered more accurate.

In that process, the application of different machine learning and deep learning techniques is crucial. Tasks such as the translation and classification of legal documents, contract reviews as well as the summarization of those are highly relevant. Deep Learning [START_REF] Lecun | Deep learning[END_REF][START_REF] Soniya | A review on advances in deep learning[END_REF][START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF] is a specific sub-field of Machine Learning, which is a specific subset of Artificial Intelligence. The concept of deep learning first emerged around 2006. Deep learning is a form of hierarchical learning and involves multiple layers of nonlinear processing for learning high-level abstractions in data [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Bengio | Learning deep architectures for AI[END_REF]. Deep learning is proving to be the next breakthrough in the field of Artificial Intelligence. With state-of-the-art results in solving a wide variety of complex tasks especially related to pattern recognition, image processing and automatic speech recognition, the area promises to hold positive results for further research. Deep learning can be performed as supervised as well as unsupervised learning. The breakthrough in the distributed representation of words using deep learning solidifies the basis of semantic analysis. Many different unsupervised training methods, which generates word embeddings from unstructured data, make the upcoming high-level semantic analysis models achieve the state-of-art results. Deep learning is penetrating its roots in every possible domain and legal domain is also receiving the aforementioned benefits. A lawyer needs to spend hours and hours on searching for relevant material and preparing arguments with relevant precedents. Artificial intelligence enables the human lawyer to work speed and more data. This show us that cooperation of human and AI is important. It aims at providing lawyers more consultancy and getting rid of fatigue duty. This review exclusively covers the recent works employing deep learning models for legal domain and suggests future research directions.

2

Research Methodology

Literature Selection

We performed an organized review of deep learning works for legal domain. The effective search includes, Journal of Machine Learning (Springer), Journal of Artificial Intelligence and Law (Springer), Nature Scientific Reports, IEEE Conference on Knowledge and Systems Engineering, ACM Conference on Knowledge Discovery and Data Mining, and the International Conference on Artificial Intelligence and Law. We searched using the combinations of keywords from "deep learning," "neural networks," "legal data," "judgments," and "cases." We limited our search to recent papers published between January, 2015, and February, 2019, and found total of 78 articles. After going through the title and abstract of all the papers, we limited our study to 14 articles that were studied with full text and further reviewed for the survey.

Research Questions

With this research we aim to address the following research questions: RQ1. What are the available legal datasets to work upon? RQ2. What are the activities for legal aspects that have been explored using deep learning? Using this analysis, researchers can identify the best suited deep learning models to work upon a specific legal task.

RQ3. What are some other activities for legal domain that are still unexplored using deep learning techniques?

Literature Review

This section presents a brief discussion on different legal tasks that have been implemented with the help of deep learning models. After reviewing the selected articles, we divided the application of deep learning to legal domain into three broad categories: legal data search, legal text analytics and legal intelligent interfaces as shown in Fig. 1. The first category includes various models developed for retrieving and classifying relevant legal text. The second category includes tasks that require NLP analysis such as summarization, case prediction, identifying sections in legal documents, translation, element extraction from documents. The third category focuses on systems developed to support legal tasks such as question-answering systems, judgment prediction systems and dialogue systems.

Legal Data Search

A legal domain specific information retrieval system was implemented by Sugathadasa et al. [START_REF] Sugathadasa | Legal document retrieval using document vector embeddings and deep learning[END_REF]. Authors implemented three different models which incorporated vector space representations of the legal domain. The first model was developed using Node2vec algorithm, second model used sentence similarity and the third was generated using a vector space from both the models and implemented using neural network. Authors concluded that the ensemble model showed higher accuracy level. As further extension, authors concluded that the approach can be used to build information retrieval systems for other domains. Traditional full text search systems finds exact match to a given string and do not take into consideration synonyms and other related terms for each word in the search string. Landthaler et al. [START_REF] Landthaler | Extending Full Text Search for Legal Document Collections Using Word Embeddings[END_REF] worked on an information retrieval system for legal domain that searched for not only the exact matches but also semantically related patterns for any arbitrary length of search query.

The system was build using word2vec implementation of word embeddings. As suggested by authors, the system can be further improved by applying various text pre-processing steps such as stemming, stop-word removal, POS tagging and others.

An automated legal document classification model, Supreme Court Classifier (SCC) was implemented by Undavia et al. [START_REF] Undavia | A Comparative Study of Classifying Legal Documents with Neural Networks[END_REF] Authors compared a number of machine learning algorithms with the recent NN-based systems. Authors evaluated their system using the Washington University School of Law Supreme Court Database

Legal Data Analytics

Elnaggar et al. [START_REF] Elnaggar | Multi-Task Deep Learning for Legal Document Translation, Summarization and Multi-Label Classification[END_REF] proposed the application of multi-task deep learning model to perform summarization, classification and translation of German legal documents using a single model. Authors suggest that due to the scarcity of German legal documents, a single model was created using the dataset and was used to transfer learning for multiple tasks. Authors concluded that the multi-task Deep learning model outperformed the state of the art results in all three tasks.

A detailed investigation of distributional representations of words and sentences, and the related machine learning and deep learning techniques was done by Wang in his thesis [START_REF] Wang | An Unsupervised Approach to Relatedness Analysis of Legal Language (Master's thesis[END_REF]. Author proposed an innovative approach, Word2Sent, for measuring the degree of similarity between sentences. Based on the results, author concluded that the domain-specific work embedding gives better results for the datasets in the domain. An approach based on LSTM model was given by Li et al. [START_REF] Li | Evaluating the Rationality of Judicial Decision with LSTM-Based Case Modeling[END_REF] for evaluating the rationality of Chinese Judicial decisions. Authors proposed a novel metric, judgment deviation, to measure the likelihood of a certain case's misjudgment. LSTM model was implemented to extract the elements that effect the decision. Experiments were carried out on Chinese judgments taken from China Judgments online and validation results were satisfactory.

A study on recognizing logical patterns in Vietnamese legal dataset was done by Son et al. [START_REF] Son | Recognizing logical parts in legal texts using neural architectures[END_REF] using deep learning models. Authors performed experiments using four models based on recurrent neural networks including Long Short Term Memory (LSTM), Bidirectional LSTM and their combination with Conditional Random Fields. Experiments showed that neural networks approaches achieved promising results for this task. Chalkidis et al. [START_REF] Chalkidis | A Deep Learning Approach to Contract Element Extraction[END_REF] developed contract element extraction system using deep learning method. Authors implemented a Bi-LSTM model operating on word, POS tag, and tokenshape embeddings. The system was evaluated using the dataset of 3,500 English contracts having 11 categories of contracts. Authors suggest that by stacking an additional LSTM on top of the Bi-LSTM, or by adding a CRF layer on top of the Bi-LSTM, results were further improved. Authors in their work [START_REF] Neill | Classifying sentential modality in legal language: a use case in financial regulations, acts and directives[END_REF] compared deep learning architectures with traditional algorithms ranging from SVM to ensemble-based decision tree classifiers. Authors present a deep learning architecture for classifying deontic modalities in legal texts. Neural network based classifiers especially LSTM model showed consistent improvement over other classifiers. Authors conclude that further extension is possible by working on other domains.

Legal Intelligent Interfaces

John et al. [START_REF] John | Legalbot: A Deep Learning-Based Conversational Agent in the Legal Domain[END_REF] worked on a conversational system 'legalbot' for legal domain. The system responded to user queries posted as questions. Instead of going for a retrieval based system authors proposed a generative model. The model was build using the Seq2Seq deep learning model. The proposed generative system makes use of domain specific knowledge for generating answers. The system was trained using dataset build from question-answers on some legal concerns. Authors concluded that the results were promising and can be further improved by increasing the dataset provided to the model. Another legal question-answering system was given by Do et al. [START_REF] Do | Legal question answering using ranking SVM and deep convolutional neural network[END_REF]. The system was build using ranking SVM and convolution neural network. Authors suggest that characteristics of legal text such as references between articles or structured relations in sentences can be explored further to improve the obtained results.

A deep learning based prediction system was proposed by Kowsrihawat et al. [START_REF] Kowsrihawat | Predicting Judicial Decisions of Criminal Cases from Thai Supreme Court Using Bi-directional GRU with Attention Mechanism[END_REF] for decision of criminal cases. Authors implemented a Bi-directional GRU based decision system for Thai Supreme Court. Earlier systems were build based on bag-ofwords model, which generally had a low accuracy as the order of word occurrence is not considered. Recurrent neural networks was implemented to read the fact from an input case and then attention mechanism was used to compare them against relevant legal provisions. The model's output shows if a person is guilty of a crime or not. The proposed system produced a better F1 score than Naïve Bayes and SVM classification.

Table 1. gives a summary of the legal tasks, approach and the legal dataset on which the approach was validated.

Conclusion

The use of deep learning and other AI techniques in legal services will accelerate the overall process of judiciary system. The application of deep learning models in various tasks such as legal data search, predictive systems, information retrieval, extraction of relevant text, intelligent interfaces, and legal conversational agents will reduce time, effort and overall cost involved in the domain. From the study, we come to following results:

• Classification of documents is majorly implemented using convolutional neural networks and its variants. Information retrieval systems are enhanced by building domain-specific word embeddings.

• Legal text analytics involving summarization, extraction of relevant text and translation is mostly performed using LSTM models, a variant of recurrent neural network.

• To work on intelligent systems, generative models from deep learning are implemented and providing good results.

• From the datasets, it is also revealed that a number of countries are trying to use deep learning to improve their judicial systems.

We conclude that the application of deep learning in legal domain has accelerated in last two years, and thus the research is under its initial phase. The comparative evaluation for our survey was not possible as the datasets used in each of the works is unique. The area holds promising future scope, as some other tasks like context-based summarization, predicting the time that will be required to solve a case, and other legal problems can be further explored with the application of suitable deep learning techniques.

Fig. 1 .

 1 Fig. 1. Categorization of legal activities

  Wei et al.[START_REF] Wei | Empirical Study of Deep Learning for Text Classification in Legal Document Review[END_REF] reports preliminary studies in using deep learning for text classification in legal document review. Experiments were conducted on four legal datasets wherein authors compared results of neural network with SVM algorithm. Results showed that CNN gave better accuracy with training dataset of larger size and can be further improved for the text classification in legal industry. A classification system for Brazilian court's document was implemented by Wei et al.[START_REF] Silva | Document type classification for brazil's supreme court using a convolutional neural network[END_REF]. Authors implemented CNN network and obtained satisfactory results.

	(SCDB). CNN network with word2vec vector performed best and gave an accuracy
	around 72.4%.
			Retrieval
		Legal Data Search
	Deep Learning in Legal	Legal Data Analytics	Classification Relevant Text Extraction Text Summarization Text Translation Conversational Systems
		Legal Intelligent Systems
			Prediction Systems

Table 1 .

 1 Summary of Legal Tasks, Approach and the legal Dataset

	Legal Task	Sub-Category	Approach	Dataset	Results
	Category
	Legal Data	Document	Word embeddings	2500 legal cases	NA
	Search	Retrieval	with neural network [6]	from Findlaw website [9-from paper]
			Word	EU-DPD German	NA
			Embeddings [7]	rental contracts
		Document	Recurrent Neural	University School	word2vec + CNN
		Classification	Network and	of Law Supreme	achieves 72.4%
			Convolution	Court Database
			Neural Network	(SCDB)
			[8]
			Convolution	Four real legal	CNN
			Neural Networks	datasets	outperformed
			[9]		SVM