Thinh D Bui

Son T Nguyen

Quoc B Ho
email: hbquoc@fit.hcmus.edu.vn

Towards a Conceptual Search for Vietnamese Legal Text

Keywords: legal text processing, legal text mining, legal ontology, information retrieval

In this paper, a system of search engine is built based on documents of Vietnamese legal system. The key factor of this system is the ability of indexing documents in several aspects: not only on the whole text but also on logical structures and ontologies of Vietnamese legal documents. There are two important phases in the system; rstly, focusing on the recognition of the structure of Vietnamese legal text and the extraction of ontology of documents stored in the database of Vietnam Ministry of Justice; secondly, building an automated information retrieval system for the advanced search on demand of Vietnamese citizens. We study Vietnamese legal domain in both linguistic features and patterns of recognition. Experimental result on recognizing logical structures got of 64.37% on assumption annotation, 64.15% on provision annotation and 75.76% on sanction annotation in the F β=1 score on the pre-built corpus of Vietnamese Enterprise Law articles. We also evaluated the precision of classifying ontology concepts and got 86.4 % in result on testing sample set of Vietnamese legal ontology. The goal is that a search engine is proposed with indexed logical and semantic properties based on works of the rst phase.

Introduction

Legal texts are a kind of document with specic characteristics, which is dierent from other daily-used documents due to their length and complexity [START_REF] Bach | Rre task: The task of recognition of requisite part and eectuation part in law sentences[END_REF] [START_REF] Nguyen | Supervised and semi-supervised sequence learning for recognition of requisite part and eectuation part in law sentences[END_REF].

It is very important to be able to structure and extract information automatically from legal texts containing legal articles and cases in order to meet two clearly predened goals [START_REF] Nakamura | Towards translation of legal sentences into logical forms[END_REF]; rstly, supporting experts to establish complete and consistent laws and secondly deploying information systems which help citizens understand laws more eectively.

In syntactic aspect, our study of Vietnamese legal texts leads to a judgment that the law sentences and paragraphs have some specic structures [START_REF] Bach | Rre task: The task of recognition of requisite part and eectuation part in law sentences[END_REF] [START_REF] Nguyen | Supervised and semi-supervised sequence learning for recognition of requisite part and eectuation part in law sentences[END_REF]. In most cases, an article which is compound by one or many sentences can regularly be divided into two of three parts: the assumption part, the provision part, and the sanction part [START_REF] Duong | General Theory of State and Law[END_REF]. Recognizing the logical structures of legal articles is an important task to understand the meaning of legal domain and a preliminary step in problems such as translating a legal article into the logical form, legal text summarization, and question-answering in this domain [START_REF] Nakamura | Towards translation of legal sentences into logical forms[END_REF]. In a legal article, the assumption part usually describes the subjects and cases in which the law can be applied, and the provision part tells mandatory regulations related to the subjects. The sanction part describes a forceful penalties in cases the subjects contravene the provision which is applied to them [START_REF] Duong | General Theory of State and Law[END_REF]. Therefore, the outputs of recognition of logical structures of legal text is very helpful to citizens who want to understand the law: what a legal article says, or which cases the law can be applied, or which subjects related to provision or sanction described in the legal article [START_REF] Bach | Rre task: The task of recognition of requisite part and eectuation part in law sentences[END_REF] [START_REF] Nguyen | Supervised and semi-supervised sequence learning for recognition of requisite part and eectuation part in law sentences[END_REF].

Moreover, since legal domain is highly entangled with common sense views on the nature of social events, roles, and actions, the task of building legal ontologies is essential to cover the understanding of these concepts and to help legal documents be machine readable. We presently aim to build a fullled Enterprise Law ontology on a schema as a subtree of LKIF core ontology schema [5][6].

In this paper, we present a conceptual search for Vietnamese legal text which use logical and semantic annotations. We also implement a system based on an open-source search engine and evaluate this system by Precison at k method [START_REF] Manning | Introduction to information retrieval[END_REF].

The remains of this paper are organized as follows. Firstly, Section 2 present the related work on legal texts. Next, we describe the conceptual search and an information retrieval (IR) system implemented on a database of Vietnamese Enterprise legal documents in Section 3. Then, Section 4 presents some experiments on each phase of our system. Finally, conclusion and future work are presented in Section 5.

Related works

According to our study of Vietnamese Law, Vietnamese legal documents have been composed carefully by experts, include a large amount of clear concepts and have some specic structures [START_REF] Duong | General Theory of State and Law[END_REF]. The language presented in these legal documents is Vietnamese written language which is used ocially by the Vietnamese Government to ensure the seriousness, accuracy and consistency of these documents. As experts recommend, a law article regularly be analyzed into logical part annotations including the assumption part, the provision part, and the sanction part [START_REF] Duong | General Theory of State and Law[END_REF]. Our view on the recognition of logical structure of Vietnamese Law is closed to the latest research of authors in [START_REF] Bach | Rre task: The task of recognition of requisite part and eectuation part in law sentences[END_REF] [START_REF] Nguyen | Supervised and semi-supervised sequence learning for recognition of requisite part and eectuation part in law sentences[END_REF]. There might be some differences in the structure of legal documents of each nation but the background, motivation and target is the same.

In [START_REF] Bach | Rre task: The task of recognition of requisite part and eectuation part in law sentences[END_REF], the authors divided a law sentence into two parts: a requisite part and an eectuation part. These two parts again are composed of three parts: a topic part, an antecedent part, and a consequent part and there are 4 cases have been made from this composing. The authors presented how to model the task as a sequence learning problem. The model is experimented on an annotated corpus of Japanese national pension law sentences. Then, with the logical part annotations along with using natural language processing (NLP) techniques, the eld of AI and Law Legal Engineering can address a range of problems aiming to achieve a trustworthy electronic society [START_REF] Nakamura | Towards translation of legal sentences into logical forms[END_REF].

While legal documents are always concerned with constraining and controlling social activities using documented norms, they are still lack of ontologies, in comparison with other domains such as medicine, engineering, or psychology [START_REF] Breuker | The construction and use of ontologies of criminal law in the ecourt european project[END_REF].

In [START_REF] Breuker | The construction and use of ontologies of criminal law in the ecourt european project[END_REF], the authors claimed that legal ontologies are about identifying concepts rather than recognizing any types of knowledge or reasoning roles. Any foundation ontology including notions about agent, actions, time, space . . . should be constructed to be able to model and understand legal domain. Although there were some available foundation ontologies such as the CYC upper ontology and the IEEE-Standard Upper Ontology (SUO) but they are not suciently covering this domain []. The core ontologies about law can be mentioned such as FOLaw and followed by LRI-core [5][6]. In particular, the LRI core presents three layers of abstraction: the foundation ontology containing general concepts, the core-ontology containing typical concepts for law, and the specic domainontology (enterprise, criminal law as case studieds) [START_REF] Breuker | The construction and use of ontologies of criminal law in the ecourt european project[END_REF]. Next to LRI core, a legal core ontology called Legal Knowledge Interchange Format (LKIF) is developed in Estrella project, contains basic concepts for law with a considerable number of terms [START_REF] Hoekstra | Lkif core: Principled ontology development for the legal domain[END_REF].

Until now, there are several approaches to ontology design and development mentioned in [START_REF] Francesconi | Integrating a bottom-up and top-down methodology for building semantic resources for the multilingual legal domain[END_REF][6]: top-down approach, bottom-up approach and the hybrid approach called "middle-out". With the logical part annotations and ontology annotations along with using natural language processing techniques, the eld of AI and Law can address a range of problems such as translating a legal rule into a logical form, question answering in legal domains, automated knowledge extraction from legal texts [START_REF] Nakamura | Towards translation of legal sentences into logical forms[END_REF].

In Vietnam, it is still lack of legal domain ontology since the legal processing has not been started. The search engines on Vietnamese legal documents just simply treat legal documents as normal text. Based on our knowledge, it is the rst research on semantic aspects of Vietnamese legal text.

A Conceptual Search for Vietnamese Legal Text

Our conceptual search for Vietnamese legal text is proposed based on results of two main tasks: structuring legal texts and building legal domain ontology. This search is deployed as an IR system. We rstly present the overall architecture of IR system in Section 3.1. Then, two main tasks would be presented in Section 3.2 and Section 3.3.

IR System Architecture

The architecture of proposed IR system has two isolated phases which are shown in Figure 1.

In the rst phase, there are two main components as follows:

Structure Annotator

Semantic Annotator

Legal Ontology

Indexer

Index file

Query Processor

Top-k retrieval

Legal documents

User Query (2) Semantic annotator: is consider as one component which takes responsibility for classifying the candidate terms and populating them into high level abstract concepts on our ontology schema, a subtree of LKIF core ontology schema [START_REF] Hoekstra | Lkif core: Principled ontology development for the legal domain[END_REF] [START_REF] Breuker | The construction and use of ontologies of criminal law in the ecourt european project[END_REF]. An Enterprise Law ontology is gradually fullled and their instances would be matched backwards to documents to annotating them.

Result

Search engine

In the second phase, a search engine using Apache Lucene libraries 1 will be implemented based on the output of the previous phase. In particular, it would be a collection of annotated documents. An Indexer is used to index this collection and create index les on several elds: full content eld, logical structure elds, and semantic concept elds.

The rest of the system is how the Query processor works on the input queries. Each query will be analyzed into a set of keywords (tokens). Next, proximity search method is used to increase the hits of relevant documents. Note that Top-k retrieval method with ranking score computed at Lucene matching model will be used also. A graphics interface is implemented to show out the results to user.

We should say that the core of this system is two main tasks in the rst step that are describe below: The structure of legal documents in Vietnamese legal system can be described as follows: each document contains several chapters and each chapter includes at least one article. In some cases, a document contains only articles without any chapters. While analyzing legal articles which are the core of Vietnamese legal documents, we realize that an article is regularly divided into three types of logical part in dierent orders: the assumption part, the provision part, and the sanction part [START_REF] Duong | General Theory of State and Law[END_REF].

In the proposed approach, we study the sample patterns from Vietnam legal documents through analyzing their linguistic features. In particular, we use GATE framework [START_REF] Cunningham | A framework and graphical development environment for robust nlp tools and applications[END_REF] along with some NLP plugins such as ANNIE, VietNLP 2 , Hash Gazetteer, and Java annotation patterns engine (JAPE). The input should be a Vietnamese legal text and it would be annotated with several annotation sets: chapter (< chapter >), article (< article >), assumption (< a >), provision (< p >), and sanction (< s >).

This annotating process is conducted based on the trigger sets which are transformed to hash gazetteers and the patterns written in JAPE transducer.

One sample of patterns is presented in Table 1.

# # # #

* *

The trigger sets includes ve kinds of tag as follows:

(1) adv: adverbs representing the situations of an assumption part.

(2) agent: nouns or noun phrases representing subjects and objects.

(3) provisionP ref ix: verbs usually appearing in the beginning of a provision part.

(4) provisionV erb: main verbs usually appearing in a provision part.

2 http://vlsp.vietlp.org:8080/demo/?page=resources [START_REF] Breuker | The construction and use of ontologies of criminal law in the ecourt european project[END_REF] sanctionV erb: main verbs usually appearing in a sanction part.

Note that mentioned-above provisionT rigger is composed of provisionP ref ix and provisionV erb.

The output of these steps is an annotated text with logical structure annotation sets. Some annotated samples of article in Vietnamese and English are described in Table 2.

Building Vietnamese Legal Ontology on LKIF Core Ontology Schema

The second task in the mentioned-above rst phase is semantic annotating on legal concepts with corpus of Vietnamese Enterprise Law articles. This task is done by "middle-out" approach with four following steps: Firstly, we reuse a subtree of LKIF core-ontology schema [START_REF] Hoekstra | Lkif core: Principled ontology development for the legal domain[END_REF] and do partial mapping to concepts in Vietnamese legal text, as described in Figure 2. The concepts at leaf nodes would be populated with terms. Then, the candidate terms which are extracted from the output of automatic linguistic analysis (chunking) will be selected to populated into classes of the ontology schema. This step is repeated to create a fullled ontology from Vietnamese Legal Text.

In the initial approach, noun chunks are chosen as candidate terms and we just use 5 concepts as main classes to classify terms. The rule to extract noun phrases is presented (in POS display) in Table 3.

Methods in [START_REF] Frantzi | Automatic recognition of multi-word terms: the c-value/nc-value method[END_REF] are used to select candidate terms by calculating phrases' N/NC value. The candidate ones will be classied into ve main classes by a Finally, we will put legal terms into the ontology schema to form it gradually fullled.

Corpus and Evaluation Methods

This section presents our corpus for each task belongs to proposed framework and associated evaluation methods.

3 http://www.csie.ntu.edu.tw/ cjlin/libsvmtools/

Vietnamese Legal Document Database and Training Corpus

We have implemented a semi-automated tool to collect and purge legal documents from Vietnam Ministry of Justice. A database of 83 Vietnamese Enterprise legal documents including 9379 sentences within 1997 articles is stored sustainably. Each document is stored along with its metadata such as: ocial number, issued date, validated date, legislation type, eective area, status, expired date, expired part, and altered part.

After that, according to our study of Vietnamese legal text, a golden corpus of annotated Vietnamese Enterprise Law articles is built manually and stored in XML format. Some statistics on this corpus are shown in Table 4. We have some remarks here. About 59.3% of logical part is assumption, 35.1% is provision and only 5.6% is sanction.

The Task of Recognition of Logical Structure

We directly applied rule-based approach with our own recognition patterns to experiment the corpus. The results were evaluated using precision, recall and F β=1 scores as follows [START_REF] Goutte | A probabilistic interpretation of precision, recall and f-score, with implication for evaluation[END_REF]: end word and correct annotated type [START_REF] Bach | Rre task: The task of recognition of requisite part and eectuation part in law sentences[END_REF].

Annotation Precisi on (%) Recall (%) (%
The result is shown in Table 5.

The Task of Ontology Instances Classication

On this task, there are 2733 noun chunks extracted from database of Vietnamese Enterprise Law and there are 299 candidate terms which are selected after that.

In this experiment, we implemented SVM method with LIBSVM with a set of 5 linguistic features corresponding to each candidate term: character sequence, previous word, following word, previous part-of-speech, and following part-ofspeech.

The training sample set and test sample set are built based on the above candidate terms with mentioned linguistic features. So, the size of training set reaches to 6500 samples. We also make a test set with 500 samples.

After training, the classier achieved 86.4% in precision score with 432 correct samples over 500 test samples.

Evaluation on Information Retrieval System

We evaluated our IR system with the mentioned-above database. The range of criteria used for evaluation includes: on the whole document and on logical parts of the legal article.

In the rst experiment, we used Precision at k method to identify the relative precision with proximity search function in Lucene framework to evaluate the system. The result is described in Table 6.

We have some remarks here about the correlation between Top k and word distance (in proximity search). The best precision is at case of k = 30 and distance = 1. In almost case, the precision goes down when distance increases.

So, it is not eective with proximity search in Vietnamese legal texts. As table shows, when k increases to 30, the higher word distance lead to much worse precision.

In the second experiment, we performed searching on the whole document and on logical parts of the document with associated keywords. For example, a keywords set related to assumption will be used to evaluate system on two ranges of criteria: assumption and full-text. The result in Table 7 describes the percentage of relevant documents over all retrieval documents. We should say that the result while searching on assumption part got lower precision than the result while searching on full-text. Note that we use a keywords set related to assumption here. The reason is the number of assumption is large but recognition of assumption part just got low precision.

Conclusion and Future Work

We presented a conceptual search for Vietnamese Legal text. This search is based on outputs of two main task: recognizing logical structures and building ontology of Vietnamese legal text. We implemented an information retrieval system using However, there are still some limitations in our research such as the small corpus and old methods utilized. Vietnamese legal ontology schema has been built but it is still lack of concepts. To get better result in further research, the corpus and ontology should be extended and state of the art methods should also be implemented. We also plan to extract verb phrases as candidate terms and expand the number of concepts in the ontology schema.

Fig. 1 .

 1 Fig. 1. IR System Architecture

  [START_REF] Bach | Rre task: The task of recognition of requisite part and eectuation part in law sentences[END_REF] recognizing of logical structures of Vietnamese legal text; (2) building Vietnamese legal ontology based on LKIF core ontology.

  1 http://lucene.apache.org/ 3.2 Recognition of Logical Structure of Vietnamese Legal Text Our research on Vietnamese legal documents shows that only a few fundamental structures are used. Because legal texts are a kind of highly semantic and consistent document, we decide to apply a rule-based approach to recognize logical parts of text to annotate them based on regular expressions and linguistic features of Vietnamese legal documents.

  (1) ontology schema partial reuse; (2) chunk extraction; (3) candidate term selection; (4) term classication and population.

Fig. 2 .

 2 Fig. 2. Partial mapping from LKIF core ontology to Vietnamese Enterprise Law

  this conceptual search to work on documents stored in law database of Vietnam Ministry of Justice. With the input as a database of logical part annotated and semantic annotated legal documents, the system consists two phases that aim to retrieve most relevant documents; rstly, annotating logical parts and semantic concepts; secondly, using Lucene libraries to index on annotation elds and search with proximate range. Experiments on each individual phase showed that the task of recognizing can achieve positive result in F β=1 score on Vietnamese legal article corpus and the classication model for Vietnamese Enterprise Law ontology got 86.4% in precision. The system is also evaluated by Precision at k method on other ranges of criteria. It is the baseline for future works on legal domain in Vietnam.

Table 1 .

 1 A sample of pattern in regular expression

	#
	#
	#
	#
	2 *

* Pattern of Recognition S1= ([<a> ADV VietToken* </a> stopword | " ,"])? [<a> AGENT VietToken* </a> ] <p> PROVISIONTRIGGER VietToken* </p> ([ <a> ADV VietToken* </a>])?

Table 2 .

 2 Several samples of Vietnamese legal article (and in English)

	Vietnamese	English
	<a>Đối với Dự án trong Danh mục dự	<a> As concerned with the project in
	án đã công bố có từ 2 Nhà đầu tư trở lên	the publicized project directory which
	cùng đăng ký thực hiện</a>, <a>Cơ	has been signed up to the development
	quan nhà nước có thẩm quyền</a> <p>	by at least 2 investors</a>, <a>
	phải tổ chức đấu thầu rộng rãi trong	authorized government offices </a>
	nước hoặc quốc tế để lựa chọn Nhà đầu	<p> have to give the bidding out the
	tư</p>.	domestic or overseas public</p>.
	Pattern of Recognition
	S1=
	([<a> ADV VietToken* </a> stopword | " ,"])?
	[<a> AGENT VietToken* </a> ]
	<p> PROVISIONTRIGGER VietToken* </p>
	([ <a> ADV VietToken* </a>])?

<a>Tổ chức kinh tế hoạt động dưới danh nghĩa công ty nhà nước</a> mà <a>không có quyết định thành lập</a> thì <s>bị đình chỉ hoạt động và bị tịch thu tài sản nộp vào ngân sách nhà nước</s>. <a>Business entity that operates under a state organization name </a> <a>without the establishment licenses </a> will <s> be suspended and all of its assets will be confiscated to contribute to the State Budget</s>.

Table 4 .

 4 Statistics on Vietnamese Enterprise Law rule corpus

	Type	Number		Part type	Number
	Sentence	529
	Rule	132	assumption	393
				provision	233	Vietnamese
				sanction	37	<a>Đối với Dự án trong
						Danh mục dự án đã công
						bố có từ 2 Nhà đầu tư
						trở lên cùng đăng ký
	Annotation	Precision (%)	Recall (%)	(%)	thực hiện</a>, <a>Cơ quan nhà nước có thẩm quyền</a> <p> phải tổ
						chức đấu thầu rộng rãi
	assumption	67.13		61.83	64.37	trong nước hoặc quốc tế
						để lựa chọn Nhà đầu
	provision	71.2		58.37	64.15	tư</p>.
	sanction	86.2		67.57	75.76	<a>Tổ chức kinh tế hoạt
						động dưới danh nghĩa
						công ty nhà nước</a>
						mà <a>không có quyết
						định thành lập</a> thì
						<s>bị đình chỉ hoạt động
						và bị tịch thu tài sản nộp
						vào ngân sách nhà
						nước</s>.

  part is recognized correctly if and only if it has correct start word, correct

				)
	assumption	67.13	61.83	64.37
	provision	71.2	58.37	64.15
	sanction	86.2	67.57	75.76
	ith the
	ized
	ich has	#
	e ast 2	#
	authorized	#
	/a> <p>	#
	ing out
	seas	2 *	*
	hat
	e
	/a>
	lishment
	> be
	its assets
	o
	te

A

Table 5 .

 5 Experiment result of rule-based approach

	Type	Number		Part type	Number
	Sentence	529
	Rule	132	assumption	393
				provision	233	Vietnamese
				sanction	37	<a>Đối với Dự án tron
						Danh mục dự án đã cô
						bố có từ 2 Nhà đầu tư
						trở lên cùng đăng ký
	Annotation	Precision (%)	Recall (%)	(%)	thực hiện</a>, <a>Cơ quan nhà nước có thẩm quyền</a> <p> phải tổ
						chức đấu thầu rộng rãi
	assumption	67.13		61.83	64.37	trong nước hoặc quốc
						để lựa chọn Nhà đầu
	provision	71.2		58.37	64.15	tư</p>.
	sanction	86.2		67.57	75.76	<a>Tổ chức kinh tế ho
						động dưới danh nghĩa
						công ty nhà nước</a>
						mà <a>không có quyế
						định thành lập</a> thì
						<s>bị đình chỉ hoạt độ
						và bị tịch thu tài sản nộ
						vào ngân sách nhà
						nước</s>.

Table 6 .

 6 Experiment result in case searching on the whole document

			Top k
	Word	K=10	K=20	K=30
	distance	(%)	(%)	(%)
	1	56.00	62.14	71.07
	2	54.00	59.93	70.00
	3	60.00	57.75	59.69
	4	61.00	58.10	59.05
	5	61.00	57.18	59.64

Table 7 .

 7 Experiment result in various range of criteria

					Top k					Top k
	Word distanc e	K=10 (%)	K=20 (%)	K=30 (%)	K=40 (%)	K=50 (%)	e Word distanc	K=10 (%)	K=20 (%)	K=30 (%)
	1	56.00	62.14	71.07	71.09	68.00	1	56.00	62.14	71.07
	2	54.00	59.93	70.00	71.09	68.00	2	54.00	59.93	70.00
	3	60.00	57.75	59.69	71.09	68.00	3	60.00	57.75	59.69
	4	61.00	58.10	59.05	64.05	68.00	4	61.00	58.10	59.05
	5	61.00	57.18	59.64	63.87	68.00	5	61.00	57.18	59.64
							Keyword set
					Assumption	Provision	Sanction
		Range of criteria	Assumption Provision Sanction Full-document	80% 85%	63.33% 62.22%	64.67% 52%

Acknowledgment

This work is supported by Vietnam National University, Ho Chi Minh City. We would like to thank Akira Shimazu and Nguyen Le Minh who has recommended this research topic to our research group at Information System department.


Huu-Thanh Duong

Bao-Quoc Ho

A Vietnamese Question Answering System in Vietnam's Legal Documents

Keywords: Question Answering, QA, Legal Document, Law on Enterprises, Information Retrieval, IR, Natural Language Processing

In this paper, we develop a Vietnamese Question Answering system to answer simple questions about provisions, processes, procedures, and sanctions in law on enterprises Vietnam. Research to build a Vietnamese Question Answering system is more difficult than English Question Answering system because of lack of Vietnamese processing resources and tools, or their results isn't high. We have utilized available Vietnamese resources, tools and modified some tools (such as Lucene) and algorithms, which worked well in English, applies in Vietnamese. From that, we have proposed a similarity-based model to build a Vietnamese Question Answering system in Vietnam's legal documents, namely vLawyer system. In experimental section, we achieved promising result, about 70% precision in legal documents, it proved that our approach is reasonable in legal document domain.

Introduction

Today, the development of the information technology has made people overwhelmed in huge documents in the internet. The internet contains lots of useful documents for users, but it has also no less dirty ones. So, to utilize useful documents quickly and effectively, it has to be necessary to develop a system to satisfy those demands. A Question Answering (QA) system is the answer for that. It isn't like the information retrieval (IR) system, only returning a list of documents related to keywords of user's query, QA gives a succinct and short answer for a query as user's natural language. QA becomes a subject which has attracted lots of research groups. It is a system inherited from the information retrieval system, it helps text mining becomes more effectively, quickly and practically. While the English QA achieved lots of remarkable results, the Vietnamese QA has still been under development because there are lots of difficulties in lack of resources and tools to process natural language such as Chunker, Named Entity Recognition (NER), etc. In this paper, we want to utilize some available tools in Vietnamese natural language processing such as VietTokenizer, jvnTagger, etc and other tools such as Lucene which is a very strong tool to support to build information retrieval systems. Besides, we have also utilized ideas of some algorithms applied in English effectively such as KEA (Keyphrase Extraction Algorithm) to identify the keyphrases, this algorithm is cross-language. With limited resources and tools, we hope to have a little contribution to build a Vietnamese QA system. In this system, we develop a Vietnamese QA system bases on Vietnam's legal documents, especially as law on enterprises 2005.

The remaining of this paper is organized as followed: in next section, we present some related works for our system. In section 3, we introduce to overview about legal documents. And in section 4, we present our built system, namely vLawyer, including the theoretical basis to build our system and system architecture. Finally, in section 5, we present a case study, give the conclusion and future works.

Related Works

While the IR system had more and more exact results, the QA system has not still impressed results yet, especially as Vietnamese. However, developing Vietnamese QA system has had first-step respected results. We will show some achievements in developing Vietnamese QA system as followed:

Vu Mai Tran, Vinh Duc Nguyen, Oanh Thi Tran, Uyen Thu Thi Pham, Thuy-Quang Ha [START_REF] Vu Mai Tran | VietNam: An Experimental Study of Vietnamese Question Answering System[END_REF] studied to build a Vietnamese QA system in travel domain by incorporating Snowball methodology and relational extraction using search engines. The system, firstly, extracted patterns and tuples into dataset. This system supported to answer simple questions about entities relationship. When it received a user's query, it would extract entities and look for related candidate relations. Then, it calculated similarities between query vector and all patterns of candidate relations to choose the best pattern. Finally, it built a query to database by using entity information (E) and relation (R) to get tuples contained relation R and entity E. According to authors, this system achieved 89.7% precision and 91.4% ability to give the answer when testing on traveling domain.

Mai-Vu Tran, Duc-Trong Le, Xuan-Tu Tran, Tien-Tung Nguyen [START_REF] Tran | University of Engineering And Technology: A Model of Vietnamese Person Named Entity Question Answering System[END_REF] built a Vietnamese person named entity question answering model. It's a closed QA answering person factoid questions (such as Who, Whose, Whom). The question processing model used CRF mechine learning algorithm and two automatic answering strategies: indexed sentences database-based and Google search engine-based. If answers were not found in database, the question would be pushed into Google search engine. The system filtered candidate answer collection based on their similarities with question and assigned a priotity number to the candidate answers. Finally, the system ranked the answers and sent to user for final validation in order to extract the exact answer. According to authors, the system obtains 74.63% precision and 87.9% ability to give the answer.

Dai Quoc Nguyen, Dat Quoc Nguyen, Son Bao Pham [START_REF] Dai Quoc Nguyen | College of Technology Vietnam National University, HanoI: A Vietnamese Question Answering System[END_REF] proposed an ontoloy-based Vietnamese question answering system that allowed users to make a query in natural language. It included two components: question processing and answer retrieval. They built a set of relations in the ontology that included only two person relations. According to authors, results were relatively high, the Question Analysis module and the Answer Retrieval module achieve an accuracy of 95% and 70% respectively, 5% remaining errors are due to the lack of coverage of their Jape grammars in the patternmatching module. However, the cost for building the database was high.

Dang-Tuan Nguyen and Ha Quy-Tinh Luong [START_REF] Dang Tuan | Document Searching System based on Natural Language Query Processing for Vietnam Open Courseware Library[END_REF] built a system to search courses in Vietnam OpenCourseWare program, it allowed users to input question in vietnamese natural language. It, firstly, extracted terms from VOCW pages and stored in ontologybased knowledge. When it received user's query, the system would analysis question parser based on the set of defined syntax rules and build a query in SPAEQL to get data on VOCW ontology.

Dang Truong Son, Dao Tien Dung [START_REF] Truong | Apply Mapping Question Approach in building the uestion answering system for Vietnamese language[END_REF] built a Vietnamese QA system by mapping user's query with questions in a defined dataset as couples of question/answer which were extracted from the internet. With returned answers, authors hope that one of them will satisfy user's demand. According to authors, they extracted 90000 couples of question/answer from Yahoo Answer and also got 50 questions randomly for testing, they obtained higher results compared with answering results from Yahoo.

From these systems, we try to build a question answering system in a relatively popular and useful domain, namely legal documents, based on similarities between documents with using tf × idf weight and latent semantic indexing.

Vietnam's Legal Documents

In this section, we introduce overview of Vietnam's legal documents and a legal structure. According to Law No. 17/2008/QH12: "Legal documents are documents issued or jointly issued by state agencies in accordance with the authority, formats, sequence of steps and procedures prescribed in this Law or the Law on the Promulgation of Legal Documents of People's Councils and People's Committees, which includes common rules of conducts, which has compulsory effectiveness and the implementation of which is guaranteed by the Government to regulate social relations.". The system of legal documents includes (in order of legal priority): A legal document content is divided by levels: chapters, sections, articles, clauses of an article, each clause can contain small sentences. When extracting legal documents from Ministry of Justice's website, the system also gets these documents and bases on this priority to assign a boost value to documents when indexing documents with Lucene. A legal structure includes: assumption, provision and sanction.

 Constitution,
 Assumption designates situations (conditions and circumstances) which can occur in practice.  Provision is subject's behaviors when it faces the situation which is mentioned in assumption section.  Sanction gives penalities if the subject does not perform rightly what the provision section designated in the situation of the assumption section.

When extracting candidate passages, the system will get an article or a clause of the article or a sentence of the clause of the article, namely a legal.

vLawyer System

We named the system as vLawyer (Virtual Lawyer), this is a Vietnamese question answering system to answer simple questions about Vietnam's law on enterprises text. Questions can mention about provisions, processes, procedures and sanctions in law on enterprises. The system allows a question as natural language and the answer is an relevant article or a clause of the article or a sentence of the clause of the article.

Theoretical basis Document Preparation.

A legal document must be concise, clear and conform. Thus, it's necessary to choose reliable sources to extract. We choose Vietnam Ministry of Justice's website to extract laws, decrees, circulars, decisions that are still validity or partially validity. All of documents were converted to lowcases and tokenized and tagged Pos by VietTokenizer and jvnTagger [16] respectively. This corpus will be bigger and bigger during user's searching process. If the system doesn't find the answer in current corpus, it will push query to Google search engine to get more documents from Vietnam Ministry of Justice's website.

At present, we extracted about 113 documents, including laws, decrees, circulars, decisions related to law on enterprise.

Lucene Tool.

Lucene [START_REF] Gospodnetíc | Lucene in Action[END_REF] is a free and strong open-source information retrieval software library, developed by Dough Cutting. This library includes basic functions for indexing and searching. Lucene responses results quickly from a big set of files because Lucene searching bases on documents' index, but not simple to search on text directly. Lucene has chosen by lots of researchers because it's simple, effective and cross-language.

In Lucene, the unit of searching and indexing is documents. A document has lots of fields, each field is a couple of name/value. Indexing in Lucene is to create the document with one or lots of fields. To index the text, Lucene extracts the keywords and removes stopwords, keywords are created inverted index and stored into each segment for searching.

After indexing documents, we can search with Lucene based indexed results. Searching in Lucene bases on a list of keywords and logic operators (AND, OR, NOT). When searching, users must designate a field to search and make a query, Lucene uses this query to search results for user, based on ranking the similarity scores between documents and the query.

In this system, we use the set of documents extracted from Vietnam Ministry of Justice's website and they are tokenized by VietTokenizer and indexed by Lucene. When the system receives user's query, it will extract keyphrases/keywords and search relevant documents by using Lucene.

Term weight.

Three following weights are used the most in term weights:  Term Frequency (tf): A number of times of a term appear in a document. It gives measure of importance of the term within the particular document.  Document Frequency (df): A number of documents contain a term. It gives measure of distribution of the term in document collection.  Collection Frequency (cf): The sum of term frequency of a term appears in all of documents of the collection.

tf×idf weight:

 Inverse Document Frequency (idf) is a measure of general importance of the term. We can use the following formular to calculate idf:

𝑖𝑑𝑓 = 𝑁 𝑑𝑓 𝑡
where N is the number of documents in collection, dft is document frequency of term t.

 tf × idf meanings: If the term appears lots of times within a document (tf is high), that term is able to be a keyword in that document. However, if that term also appears lots of times in various documents (df is also high), then it may be a common term (less meaningful) such as "và" ("and"), "hoặc" ("or"), .... When idf has responsible of descending the score of that term. This weight can be calculated as followed:

𝑤(𝑡, 𝑑) = { (1 + log(𝑡𝑓 𝑡,𝑑 ))𝑙𝑜𝑔 𝑁 𝑑𝑓 𝑡 , 𝑖𝑓 𝑡𝑓 𝑡,𝑑 ≥ 1 0, 𝑖𝑓 𝑡𝑓 𝑡,𝑑 = 0
Latent Semantic Indexing (LSI).

LSI is an approach to index automatically by mapping documents and terms into LSI space. The mapping way bases on a concept, namely Singular Value Decomposition (SVD). SVD is the way to reduce multi-dimension data, reduce a big data so that the remainder only focuses on necessary data from original data. LSI solves two problems in vector space as synonymy and polysemy.

We perform to decompose term-document matrix into smaller matrixes. To do that, we use SVD: C = U.S.V T (where C namely a term-document matrix), where:  U matrix is a matrix which each row is a vector of each term.  V T matrix is matrix which each column is a vector of each document.  S matrix is a diagonal matrix with descending singular value, it gives importance of each dimension.

Reducing SVD: We only retain k singular value, the remainder values are assigned 0 (k < min(M, N)). When S matrix is k×k, U matrix is M×k, V T matrix is k×N, Ck matrix in LSI space is M×N. We use SVD to calculate Ck matrix. Calculating similarities on Ck matrix will be more reliable result than on C matrix.

Noun Phrase and Verb Phrase.

A Vietnamese noun phrase has overview of the structure as followed: <previous sub-section> <center noun> <post sub-section> Example: "mái tóc đẹp" (a beautiful hair), noun "tóc" (hair) is a center noun, "mái" is a previous sub-section, "đẹp" is post sub-section.

It notes that a noun phrase is able to be the lack of previous sub-section or post subsection, but unable to be the lack of the center noun.

Based on this base and refer to [START_REF] Phúc | Research on application of frequent sets and association rules to semantic Vietnamese document classification[END_REF], we chose the following patterns to extract Vietnamese noun phrases: The same as verb phrase, the overview of the structure as followed: <previous sub-section> <center verb> <post sub-section>

 Noun +
In our using scope, we defined the following patterns to extract verb phrase: Besides, when extracting noun phrases and verb phrases, we also use the following rules:

 Verb +
 A phrase cannot start or end by stopwords, also contain special characters.  A phrase has maximum length as 3.  Using idf assigns score to the phrase to give the importance of the term in collection.

Based on documents were extracted on Vietnam Ministry of Justice's website, we extracted the set of 24771 two-terms phrases (both noun phrases and verb phrases) and 10360 three-terms phrases (both noun phrases and verb phrases).

Other Resources.

In limited period of time, we also made two files manually so that the system gives results better:  The set of synonymy terms in law text.  An ontology about lines of business.

System Architecture

The vLawyer system includes two components:

 Question processing: in this one, the system will convert the question to lowcases and remove stopwords, special words in user query. It also tokenizes, assigns POS tagger to query and extracts keyphrases/keywords to build into Lucene query to search relevant documents.  Answer selection: Based on documents searched above phase, we extract articles to calculate similarities with user's query and incorporate some heuristics to choose the best answer. If the system cannot give the answer from the current corpus, it will push the query to Google search engine to find more documents to supplement in current corpus. The query will be converted to lowcases and tokenized by VietTokenizer, assigned POS Tag by jvnTagger and removed stopwords, special words and question words. Extracting keyphrases, they are noun phrases or verb phrases have:  The higher is score, the better is keyphrases.  Priority for two-words phrases. If a three-words phrase contains a two-words phrase, but the two-words phrase has a higher score than the three-words phrase, we choose the two-words phrase is keyphrases.

The remaining is keywords. Afterward, we get synonymy terms of extracted terms. We build into Lucene query to search relevant documents.

Answer Selection.

We choose first top-ten documents which Lucene evaluates the best high score and perform to extract candidate passages in those documents. The candidate passage is able to be an article or a clause of the article or a small sentence of the clause of the article. They are extracted towards the following rules: When reading an article, the system calculates the similarity between the title of the article with the query, if the result is over a defined threshold, then it will extract the whole article as a candidate passage. Otherwise, it continues to read clauses 1, 2, ... of this article and each clause does the same as above to decide to choose that clause for candidate passage or not. If not, it will read smaller sentences of clause as a), b),... the process is the same as above. If it still does not choose any candidate passages, the article will be ignored and continue to read other articles. The process's performed like that until the end of the document.

If the system cannot find any results from the set of documents returned by Lucene or extract any candidate passages from relevant documents. The query will be pushed to Google search engine to find the more relevant documents. Those documents will be supplemented to the current corpus.

If the system found the set of candidate passages, the system will build a term document matrix of candidate passages and the query, each element of that matrix is tf×idf weight of a term with respective candidate passage. Then, it calculates similarities between candidate passages and the query by cosin function and orders results by descending similarity scores.

Otherwise, the system reduces the threshold that was used to extract candidate passages and performs to choose candidate passages again. If they are found, the system will map them into LSI space and calculate similarities in this space. In this case, similarity calculation is in LSI space will be more reliable than normal vector space model.

Incorporating this result and some following heuristics, the system will choose the answer to return to user:  Choosing the first top-ten results with highest similarity scores. The higher the similarity score is between candidate passage and query, the more it is an answer.  The more the candidate passage contains the keyphrases/ keywords of the query, the more it is the answer.  If two candidate passages contain the same as a number of keyphrases/keywords, the system will choose the candidate passage of the shorter length.

From those, we build the following formula to choose answers:

Where Ca: the candidate passage; Q: the user query; K: the list of keyphrases or keywords extracted from the user's query; sim(Ca, Q): the similarity score's between the candidate passage and user query; num(Ca, K): A number terms of K which Ca contains;

: the sum of scores of terms of K which Ca contains. We will choose first top-three results of Ans(Cai) score to give answers to user.

5

Experiment, conclusion and future works

Experiment

We extracted about 113 documents, including laws, decrees, circulars, decisions. All of these documents were indexed by Lucene.

Based on these documents, we built the set of 24771 two-terms phrases (both noun phrases and verb phrases) and 10360 three-terms phrases (both noun phrases and verb phrases). They were used to extracted keyphrases or keywords of user's query.

We also build manually two resources: synonymy terms and ontology-based in law text.

To illuminate for our system model, we gives a following example: it assumes that user wants to query a question as: "Việc đăng ký thay đổi người đại diện theo pháp luật của công ty trách nhiệm hữu hạn, công ty cổ phần được pháp luật quy định như thế nào?" ("How to register the replacement of the legal representative of limited liability companies or joint-stock companies?") This query will be converted into lowercase to tokenize and tag Pos as followed: Q = "việc/N đăng_ký/V thay_đổi/V người/N đại_diện/V theo/V pháp_luật/N của/E công_ty_trách_nhiệm_hữu_hạn/N ,/, công_ty_cổ_phần/N được/V pháp_luật/N quy_định/V như_thế_nào/X" Afterward, it will be removed stopwords, special words, question words to extract keyphrases and keywords. The result as followed:

 Noun phrase: người đại_diện,  Verb phrase: đăng_ký thay_đổi  Terms: pháp_luật, công_ty_trách_nhiệm_hữu_hạn, công_ty_cổ_phần

The system continue to get the synonymy terms of above terms and build a query and push to Lucene as followed: ("người đại_diện"~1 OR "chủ_sở_hữu"~) AND "đăng_ký thay_đổi"~1 AND "pháp_luật"~ AND ("công_ty tnhh"~1 OR "công_ty_trách_nhiệm_hữu_hạn"~) AND ("công_ty_cổ_phần"~ OR "công_ty cp"~1)

With the current corpus, Lucene searches 6 relevant documents, including (in order of descending scores): Banks, score = 0.11 Then, the system will extract candidate passages from these documents, the system extracts 50 candidate passages. They will be calculated the similarity score with user query, the result as followed (in order of descending score): For testing, we got randomly 211 questions about law on enterprises and related law from reliable websites (such as Vietnam Ministry of Justice, vnexpress, ...) to test our system. We performed to match the result returned by our system and the answer of domain experts. We obtained the promising result, approximately 70% precision.

Conclusion and future works

In this paper, we presented an similarity-based approach for building the vLaywer system to answer questions about provisions, processes, procedures and sanctions in law on enterprises effectively. From [START_REF] Dai Quoc Nguyen | College of Technology Vietnam National University, HanoI: A Vietnamese Question Answering System[END_REF][9], we also tried to build a ontology about lines of business manually. However, it's still small, so the results of some questions ,which related to establish lines of business, is not high. We will develop this in the future. Compare with [START_REF] Tran | University of Engineering And Technology: A Model of Vietnamese Person Named Entity Question Answering System[END_REF] [START_REF] Truong | Apply Mapping Question Approach in building the uestion answering system for Vietnamese language[END_REF] approaches, they also based on similarity between question and candidates or between question and other defined questions to choose answers for user's query, our obtained result is promising with the set of testing from internet randomly. With obtained results, it proved that our approach is reasonable.

We recognized if the set of synonymy terms and ontology in law text is larger, the result is higher. So, in the future, we will push more semantics into the system by developing ontology-based system and building synonymy corpus in legal documents automatically. Besides, we will broaden other laws and types of question.

  laws and resolutions of the National Assembly.  Ordinances and resolutions of the Standing Committee of the National Assembly.  Orders and decisions of the State President.  Decrees of the Government.  Decisions of the Prime Minister.  Resolutions of the Justices' Council of the Supreme People's Court and circulars of the Chief Justice of the Supreme People's Court.  Circulars of the President of the Supreme People's Procuracy.  Circulars of Ministers or Heads of Ministry-equivalent Agencies.  Decisions of the State Auditor General.  Joint resolutions of the Standing Committee of the National Assembly or the Government and the central offices of socio-political organizations.  Joint circulars of the Chief Justice of the Supreme People's Court and the President of the Supreme People's Procuracy; those of Ministers or Heads of Ministry equivalent Agencies and the Chief Justice of the Supreme People's Court, the President of the Supreme People's Procuracy; those of Ministers or Heads of Ministry-equivalent Agencies.  Legal documents of People's Councils and People's Committees.

Fig. 1 .

 1 Fig. 1. System Architecture



  Decree No. 43/2010/NĐ-CP: Decree on Enterprise Registration, score = 0.43  Circular No. 01/2013/TT-BKHĐT: Circular on Business Registration, score = 0.42  Law No. 60/2005/QH11: Law on Enterprise 2005, score = 0.39  Decree No. 102/2010/NĐ-CP: Decree Detailing a number of Articles of the Law on Enterprises, score = 0.26  Decree No. 05/2013/NĐ-CP: Decree on amending and supplementing some Articles in the regulations on Administrative procedures stated in the decree No. 43/2010/NĐ-CP, score = 0.22  Decree No. 59/2009/NĐ-CP: Decree on organization and operation of commercial



  Article 43 of Decree No. 43/2010/NĐ-CP: "Registration of change of owner of single member limited liability company", score = 0.431  Article 17 of circular No. 01/2013/TT-BKHĐT: "Dossier for change of legal representatives of limited liability companies and shareholding companies", score = 0.2321  Article 18 of circular No. 01/2013/TT-BKHĐT: "Registration for change of the owner of an LLC1 because of the inheritance", score = 0.2229 Article 38 of Decree No. 43/2010/NĐ-CP: "Registration of change of legal representative of limited liability company or shareholding company", score = 0.1991  etc The system will calculate other params in (*) and apply (*). As a result, it will give an answer to user as Article 17 of circular No. 01/2013/TT-BKHĐT and two reference answers: Article 18 of circular No. 01/2013/TT-BKHĐT and Article 43 of Decree No. 43/2010/NĐ-CP.


Hien T Nguyen

Phuc H Duong
email: huuphucduong@gmail.com

Vinh T Vo

Vietnamese Sentence Similarity Based on Concepts

Keywords: Paraphrase Identification, Text Similarity, Semantic Similarity

We propose a novel method for measuring semantic similarity of two sentences. The originality of the method is the way that it explores the similarity of concepts referred to in the sentences using Wikipedia. The method also exploits Wiktionary to measure word-to-word similarity. The overall semantic similarity is a linear combination of word-to-word similarity, word-order similarity, and concept similarity. We build datasets consisting of 45 Vietnamese sentence pairs and then evaluate the method on these datasets. The results show that in the best cases, concept similarity help improving the performance of our method more than 15% point. The proposed method is language-independent and quite easy to employ. Therefore, one can readily adopt our method to measure semantic similarity for sentences written in other languages.

Introduction

We study the task of measuring semantic similarity of short texts, i.e., sentence, text segments or very short text snippets. With the development of natural language applications recently, this task has been playing an increasingly important role in plagiarism detection, question answering, machine translation, text summarization, information retrieval, etc. This is a challenging task. Considering the following two sentence pairs, two sentences in the first pair are more likely similarity in meaning even though not many words they contain are common; while all most all words in two sentences in the second pair are the same but their meanings are different.

 "She has to pass the exam." and "She must get through the exam"  "To gain admission to UCLA, you need to present an academic profile much stronger than represented by the minimum UC admission requirements below." 1and "To gain admission to Berkeley, you need to present an academic profile much stronger than that represented by the minimum UC admission requirements" 2Until now, there have been many methods proposed for measuring text similarity. Most of them use either knowledge-based or corpus-based word-to-word semantic similarity measures [START_REF] Mihalcea | Corpus-based and knowledge-based measures of text semantic similarity[END_REF], in combination with syntactic information [START_REF] Li | Sentence similarity based on semantic nets and corpus statistics[END_REF], [START_REF] Oliva | SyMSS: A syntax-based measure for short-text semantic similarity[END_REF], [START_REF] Lee | A novel sentence similarity measure for semantic-based expert systems[END_REF], [START_REF] Wenyin | A short text modeling method combining semantic and statistical information[END_REF] or string matching algorithm [START_REF] Islam | Semantic text similarity using corpus-based word similarity and string similarity[END_REF]. Some works exploited machine translation metrics [START_REF] Madnani | Re-examining Machine Translation Metrics for Paraphrase Identification[END_REF], discourse information [START_REF] Bach | Exploiting discourse information to identify paraphrases[END_REF], or graph subsumption [START_REF] Rus | Paraphrase identification with lexico-syntactic graph subsumption[END_REF] for paraphrase identification. In summary, the work in literature did not exploit concepts, i.e. named entities and common concepts, including coreference and disambiguation, for measuring similarity of short texts. A concept we mean in this work is a named entity such as a person, a location, or an organization in particular, etc. or a common concept such as computer science or information technology.

We propose a novel method to compute semantic similarity between sentence pairs. We evaluate the method on a dataset consisting of Vietnamese sentences. The method explores concepts to determine how they contribute to the performance of measuring semantic similarity of sentences. It also exploits word-to-word similarity and word-order similarity as proposed by Li et al. in [4]. An intuition shows that exploring named entities (as well as concepts) and their features is potentially in improving the performance of semantic similarity of sentences, especially for those with the same meaning containing named entities but few words in common. For example, with these two sentences "I am currently working at IBM" and "I am a developer at International Business Machines", if we only compute the similarity based on words and word-order, the similarity score may not be high as it would be even though the entities IBM and International Business Machines are the same in the contexts of the two sentences.

One of the challenging problem when dealing with Vietnamese texts is that Vietnamese is a language with a deficient natural language processing support, such as no Vietnamese WordNet or corpus like Brown Corpus of American English for measuring semantic similarity between Vietnamese words. To overcome that limitation, we exploit Wikipedia3 to recognize which concepts referred to in compared sentences exist in Wikipedia to expand the context of those sentences by different surface forms of the concepts and exploit Wiktionary4 to estimate the similarity of words.

Wikipedia is a free online encyclopedia whose content is contributed by a large number of volunteer users. It consists of a large collection of articles, each of which defines and describes a concept. In reality, a concept may have several surface forms and one surface form may be used to refer to different concepts in different contexts. In Wikipedia, many-to-many correspondence between names and entities can be captured by utilizing redirect pages and disambiguation pages. A redirect page typically contains only a reference to an article. The title of a redirect page is an alternative surface form of the described entity or concept in that article. For example, from redirect pages of the United States, we extract alternative surface forms of the United States such as "US", "USA", "United States of America", etc. A disambiguation page is created for an ambiguous surface form which may use to denote two or more entities in Wikipedia. It consists of links to articles that define the different concepts having the same surface form. Wiktionary is a free-content multilingual dictionary, designed as the lexical companion to Wikipedia and opened for volunteers to edit all the contents. It provides the meaning of vocabulary, not includes the encyclopedic informationan advantage of Wikipedia.

The contributions of this paper is three-fold as follows: (i) we propose a novel method that based on concepts for measuring similarity of Vietnamese sentences, (ii) we build a dataset consisting of 45 Vietnamese sentence pairs, each of which was estimate by human subjects if it is paraphrase or not and evaluate our proposed method on this dataset. The method exploits Wiktionary to measure similarity of words, exploits Wikipedia to identify which concepts referred to in the compared sentences existing in Wikipedia for expanding the contexts of those sentences by different surface forms of the concepts. The originality of this work is the way that our method expands the contexts of compared sentences using Wikipedia.

The rest of this paper is organized as follows. Section 2 presents related work. Section 3 presents our proposed approach. Section 4 presents datasets, experiments and results. Finally, we draw conclusion in Section 5.

Related work

There is much research on sentence similarity measurement. In [START_REF] Mihalcea | Corpus-based and knowledge-based measures of text semantic similarity[END_REF], the authors proposed a method that estimates the semantic similarity between two short texts using both corpus-based and knowledge-based similarity measures of words. Given two short text segments, the method finds for each word in the first one the most similar matching word in the second one and then the similarity between those word pairs are included in the overall semantic similarity of the two text segments. In [START_REF] Islam | Semantic text similarity using corpus-based word similarity and string similarity[END_REF], the authors introduced a method that computes the text similarity by combining a corpus-based similarity measure of words and a modified version of the Longest Common Subsequence string matching algorithm. The method proposed in [START_REF] Fernando | A semantic similarity approach to paraphrase detection[END_REF] uses word-to-word similarity derived from WordNet to identify paraphrase. The methods proposed in [START_REF] Li | Sentence similarity based on semantic nets and corpus statistics[END_REF], [START_REF] Oliva | SyMSS: A syntax-based measure for short-text semantic similarity[END_REF], [START_REF] Lee | A novel sentence similarity measure for semantic-based expert systems[END_REF], [START_REF] Wenyin | A short text modeling method combining semantic and statistical information[END_REF] measure text similarity based on semantic and syntactic information contained in the compared texts. In addition to using word-to-word similarity, the method in [START_REF] Li | Sentence similarity based on semantic nets and corpus statistics[END_REF] presents the important role of word-order in improvement of sentence similarity measure, the method in [START_REF] Oliva | SyMSS: A syntax-based measure for short-text semantic similarity[END_REF] exploited adjectives and adverbs in two sentences, the method in [START_REF] Lee | A novel sentence similarity measure for semantic-based expert systems[END_REF] takes nouns and verbs in consideration, and the method in [START_REF] Wenyin | A short text modeling method combining semantic and statistical information[END_REF] exploited interdependent between word-to-word similarity and sentence similarity and computed both of them simultaneously by an iterative algorithm. In [START_REF] Rus | Paraphrase identification with lexico-syntactic graph subsumption[END_REF], the authors use graph subsumption (originally developed for recognizing entailment) with lexical, syntactic, synonymy and antonymy information to identify paraphrase.

The method proposed in [START_REF] Bach | Exploiting discourse information to identify paraphrases[END_REF] combines machine translation metrics and the ordered similarity between elementary discourse units (EDUs). An EDU are blocks of words playing an important role in sentence similarity. In [START_REF] Sahami | A web-based kernel function for measuring the similarity of short text snippets[END_REF], the authors proposed a method that takes advantage of web search results to extend context of short texts and in [START_REF] Socher | Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection[END_REF] the authors uses unfolding recursive auto-encoder method for measuring the similarity. In [START_REF] Qiu | Paraphrase recognition via dissimilarity significance classification[END_REF], instead of identifying the similarity, the authors propose a method detecting dissimilarities. The method proposed in [START_REF] Wenyin | A short text modeling method combining semantic and statistical information[END_REF] combines semantic and statistical information within short texts to compute the similarity. The method proposed in [START_REF] Lee | A novel sentence similarity measure for semantic-based expert systems[END_REF] takes advantage of corpus-based ontology to overcome the problem that evaluates the semantic similarity between irregular sentences.

In this paper, we propose a novel method that explores concepts to measure the similarity of sentences. In contrast with related methods, our method use Wiktionary instead of WordNet to measure the similarity of words. Moreover, it identifies which concepts referred to in the compared sentences existing in Wikipedia for expanding the contexts of those sentences by different surface forms of the concepts.

Proposed method

We propose a novel method that computes overall semantic similarity of two sentences, let say S1 and S2, as follows:

𝑆𝑖𝑚(𝑆 1 , 𝑆 2 ) = 𝛼 × 𝑆𝑖𝑚 𝑤𝑜𝑟𝑑 + 𝛽 × 𝑆𝑖𝑚 𝑤𝑜𝑟𝑑-𝑜𝑟𝑑𝑒𝑟 + 𝛾 × 𝑆𝑖𝑚 𝑐𝑜𝑛𝑐𝑒𝑝𝑡 (1)
where Simword, Simword-order, Simconcept are sentence similarity measures based exclusively on word-to-word similarity, word-order similarity, and concept similarity respectively; 𝛼 + 𝛽 + 𝛾 = 1; and the coefficients {𝛼, 𝛽, 𝛾} decide the contribution of wordto-word similarity, word-order similarity and concept similarity to the whole sentences. In following three sub-sections, we respectively present the similarity measures.

Model

Our method contains four main steps for computing semantic similarity between two sentences. The first step performs tokenizer. The second step calculates the similarity between sentences by using word-to-word similarity based on Wiktionary. The third computes similarity between sentences by using word-order similarity of the two sentences. The fourth step recognizes concepts in the two sentences using Wikipedia and expands the contexts of the sentences to compute the concept similarity; then the similarity between sentences is computed using the concept similarity. Finally, the overall sentence similarity score is derived by combining the word-to-word similarity, the word-order similarity and the concept similarity.

Sentence similarity based on word similarity

The essence of sentence similarity is word-to-word similarity. Thus, we propose to use the Text Overlap method to compute the word-to-word similarity between two words. To our knowledge, this is the most possible method for computing the wordto-word similarity based on Vietnamese Wiktionary. Text Overlap method was first introduced in 1986 by Michael Lesk [START_REF] Lesk | Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone[END_REF]. The main idea of the method is based on the level of the intersection of gloss texts. The higher level of intersection is, the more similar two words are and otherwise. We first have to split two input sentences into tokens, then, we will create semantic vectors base on those tokens. Now, we will go through all steps in word-to-word Sim-ilarity. Given two sentences S1 and S2, a joint word set is defined by 𝑆 = 𝑆 1 ∪ 𝑆 2 = {𝑤 1 ; 𝑤 2 ; … ; 𝑤 𝑛 }. The word set S contains all distinct words from the two sentences and includes all inflectional morphology words. For example, word and words, thesis and theses are considered as distinct words and must appear in S. For example, given the following sentences: Because the word set S is derived from two sentences, we should use it as a standard semantic vector for comparing, denoted by V. The semantic vector of two sentences (V1 and V2) must have the same dimension of V and each value of an entry of the semantic vector (V1 and V2), denoted by si (i = 1, 2, …, m), is determined by wordto-word similarity method and should lie between [0,1]. Taking S1 as an example:

 S1 = I
 Case 1: if wi appears in S1, set si to 1.  Case 2: if wi not appears in S1, word-to-word similarity method will be used to compute the similarity between wi and each word in S1. Thus, the word-to-word similarity score should be the highest number k. If k exceeds a standard threshold, then si = k, otherwise, set si = 0. Suppose that the highest number k = 0.01, the value of si should be 0, because 0.01 is closer to zero, and that's why we should have a standard threshold.

Unlike other text similarity methods, this approach keeps all function words, since these words contain syntactic information if the sentence is too short. Although they appear in the joint word set S, they can't affect the whole meaning of the sentence, as well as the semantic vectors, because we use a threshold to eliminate them. After we have two semantic vectors V1 and V2, the semantic similarity between two sentences is computed by cosine coefficient of those vectors:

𝑆𝑖𝑚 𝑤𝑜𝑟𝑑 = 𝑉 1 •𝑉 2 |𝑉 1 |•|𝑉 2 | (2)

Sentence similarity based on word order similarity

Before starting this section, we should look over the two example sentences below, they contain the same words in each sentence but differ from the position of two words boy and girl as follows:

 S1 = A boy buys the girl a gift.  S2 = A girl buys the boy a gift.

We can see that if we only consider word-to-word similarity on that candidate pair, the similarity score will be maximum because all words in that pair are the same. The dissimilarity between S1 and S2 is caused by the different word-order. To resolve this problem, we form a joint word set S, each entry value denoted by wi (i = 1, 2, …, m). S = {A, boy, buys, the, girl, a, gift}. We assign an index number for each word in S1 and S2. To compute the similarity score, a word-order vector Ri is formed for each sentence with the length equals to the size of S and each value of Ri is simply an index number in the sentence. Taking S1 as an example, for each word wi in S, we will consider its position or the most similar between it and the others, look the cases as follows:

 Case 1: if wi appears in S1, entry value of this word in R1 is the index number from S1.  Case 2: if wi doesn't appear in S1, entry value of this word in R1 is zero.

For example of S1 and S2, two word-order vectors R1 and R2 should be:

 R1 = {1, 2, 3, 4, 5, 6, 7}  R2 = {1, 5, 3, 4, 5, 2, 7}
Through this approach, we can see that R1 and R2 clearly show the basic structure of the sentence. Eventually, a measure for computing word-order similarity of the two input sentences is:

𝑆𝑖𝑚 𝑤𝑜𝑟𝑑-𝑜𝑟𝑑𝑒𝑟 = 1 - ‖𝑅 1 -𝑅 2 ‖ ‖𝑅 1 +𝑅 2 ‖ (3)

Sentence similarity based on concept similarity

In order to improve the quality of assessment of similarity between two sentences, we determine the similarity between concepts occurring in sentences base on Wikipedia.

To better understand the purpose of applying concept similarity, considering the following example with two sentences S1 and S2. If we only compute semantic similarity base on words and word-order, we can't detect two entities Ho Chi Minh City and Sai Gon are the same. As a result, it may lead to the similarity between two sentences not accurate.

 S1 = I am living in Sai Gon.  S2 = I am living in Ho Chi Minh City.

We adopt the mention recognition of Huy et al. [START_REF] Huynh | Using coreference and surrounding contexts for entity linking[END_REF] to identify surface forms of Wikipedia concepts in a sentence. If an identified surface form has only one candidate concept, our method collects all of its surface forms -extracted from its title and its redirect page titles in Wikipedia. If an identified surface form is ambiguous, our method does not disambiguate it, but collects all surface forms of all of its candidate concepts based on the disambiguation page of the identified surface form in Wikipedia.

Given a pair of sentences, let SF1 be a set of surface forms identified in the first sentence and SF2 be a set of surface forms identified in the second sentence. The sentence similarity based on concept similarity is computed as follows:

𝑆𝑖𝑚 𝑐𝑜𝑛𝑐𝑒𝑝𝑡 = |𝑆𝐹 1 ∩𝑆𝐹 2 | 𝑀𝑖𝑛(|𝑆𝐹 1 |,|𝑆𝐹 2 |) (4)

Dataset and Evaluation

Dataset

To evaluate the performance of our method, we build a dataset consisting of 45 pairs of sentences. Then the dataset is sent to different persons to estimate if each pair is paraphrase or not. If a pair is paraphrase, result will be 1, otherwise result will be 0.

After collecting seven survey results, we analyze and create two datasets to serve the assessment process. The results show that there are 19 pairs getting the same agreement by 7 persons, 6 pairs getting the same agreement by 6 persons, 6 pairs getting the same agreement by 5 persons. In total 31 pairs get the same agreement by at least 5 persons. We give 7 persons a chance to discuss on these 31 pairs and get a dataset consisting of 31 sentence pairs with agreement by 7 persons, namely Dataset-1. We build Dataset-2 as follows: the assessments which have at least four number of 1 (the same agreement by at least 4 persons) will unify to 1, the others will be 0. It means that Dataset-2 consists of 45 pairs of sentences.

Evaluation

In the fields of science, engineering, industry, and statistics, "the accuracy of a measurement system is the degree of closeness of measurements of a quantity to that quantity's actual (true) value". In our case, we use a binary classification to measure of how well a test correctly identifies or excludes a condition. Let's take a look the problem we have: the output of computational process lies in [0;1], thus, which value of threshold  give the similarity of two sentences is 1 or 0; and with that , how well the accuracy of the approach is. To solve the problem, we will initially set = 0.5, each loop  will increase a value 𝑡 = 0.01, if the output of computational process is larger than , the similarity will be 1, otherwise, will be 0. The value of 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 calculated by applying the formula:

𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = 𝑇𝑃+𝑇𝑁 𝑇𝑃+𝐹𝑃+𝑇𝑁+𝐹𝑁 (5)
where TP denotes true positive, TN denote true negative, FP denote false positive, FN denote false negative. We evaluate our method on Dataset-1 and Dataset-2 respectively. Table 1 shows the results after running our method on Dataset-1. The column "" shows the  values,

The column "accuracy with concepts" shows overall sentence similarity of our method in the term of accuracy, and the column "accuracy without concepts" shows overall sentence similarity of our method in the case we do not use concept similarity. The results shown in Table 2 are explained as the same as doing for the results shown in Table 1.

As shown in Table 1, for Dataset-1, the accuracy is maximized at threshold {0.50 to 0.55, 0.62, 0.63} and minimized at {0.94, 0.95}. As shown in Table 2, for Dataset-2, the accuracy is maximized at threshold {0.50 to 0.55, 0.62, 0.63} and min-imized at {0.94, 0.95}. Fig. 1 and Fig 2 respectively show the curves comparing the performance of our method on Dataset-1 and Dataset-2 with and without using concept similarity.

We can see that, the higher threshold is, the smaller accuracy is, because the value of  at low level is closer to people judgment. All in all, the results show that in the best cases, concept similarity help improving the performance of our method more than 15% point; which prove that concept similarity significantly contribute to the performance of our method.

Conclusion

In this paper, we present a novel method for measuring semantic similarity of two sentences. The overall semantic similarity is a linear combination of word-to-word similarity, word-order similarity, and concept similarity. The method exploits Wiktionary to measure similarity of words, exploits Wikipedia to identify which concepts referred to in the compared sentences existing in Wikipedia for expanding the contexts of those sentences by different surface forms of the concepts in order to compute the concept similarity. We evaluate the method on the datasets consisting of Vietnamese sentence pairs. The results show that in the best cases, concept similarity help improving the performance of our method more than 15% point. The proposed method is language-independent and quite easy to employ. Therefore, one can readily adopt our method to measure semantic similarity for sentences written in other languages.

Fig. 1 .

 1 Fig. 1. The curves comparing the performance on Dataset-1 with and without using concept similarity

Fig. 2 .

 2 Fig. 2. The curves comparing the performance on Dataset-2 with and without using concept similarity

  am a developer at International Business Machines.

		S2 = I am currently working at IBM.
		S = {I, am, a, developer, at, International, Business, Machines, currently, work-
		ing, IBM}.

Table 1 .

 1 The overall accuray of our method on Datatset-1

		Accuracy with Concepts	Accuracy without Concepts		Accuracy with Concepts	Accuracy without Concepts
	0.50	93.55	77.42	0.73	64.52	38.71
	0.51	93.55	77.42	0.74	64.52	38.71
	0.52	93.55	77.42	0.75	64.52	38.71
	0.53	93.55	77.42	0.76	64.52	38.71
	0.54	93.55	77.42	0.77	64.52	38.71
	0.55	93.55	77.42	0.78	61.29	38.71
	0.56	90.32	74.19	0.79	54.84	35.48
	0.57	90.32	74.19	0.80	48.39	35.48
	0.58	90.32	74.19	0.81	41.94	35.48
	0.59	90.32	74.19	0.82	41.94	35.48
	0.60	90.32	70.97	0.83	38.71	35.48
	0.61	90.32	70.97	0.84	38.71	35.48
	0.62	93.55	70.97	0.85	38.71	32.26
	0.63	93.55	67.74	0.86	38.71	29.03
	0.64	90.32	64.52	0.87	35.48	29.03
	0.65	87.1	64.52	0.88	32.26	29.03
	0.66	83.87	58.06	0.89	32.26	29.03
	0.67	83.87	58.06	0.90	32.26	25.81
	0.68	80.65	58.06	0.91	29.03	25.81
	0.69	80.65	51.61	0.92	29.03	22.58
	0.70	77.42	45.16	0.93	29.03	22.58
	0.71	74.19	45.16	0.94	25.81	22.58
	0.72	67.74	41.94	0.95	25.81	22.58

Table 2 .

 2 The overall accuray of our method on Datatset-2

		Accuracy with Concepts	Accuracy without Concepts		Accuracy with Concepts	Accuracy without Concepts
	0.50	86.67	73.33	0.73	64.44	46.67
	0.51	86.67	73.33	0.74	64.44	46.67
	0.52	86.67	73.33	0.75	64.44	46.67
	0.53	86.67	73.33	0.76	66.67	46.67
	0.54	86.67	73.33	0.77	66.67	44.44
	0.55	86.67	73.33	0.78	62.22	44.44
	0.56	84.44	71.11	0.79	57.78	44.44
	0.57	84.44	71.11	0.80	55.56	44.44
	0.58	84.44	71.11	0.81	51.11	44.44
	0.59	84.44	71.11	0.82	48.89	44.44
	0.60	84.44	68.89	0.83	46.67	44.44
	0.61	84.44	68.89	0.84	46.67	44.44
	0.62	86.67	68.89	0.85	44.44	42.22
	0.63	86.67	68.89	0.86	46.67	40.00
	0.64	82.22	66.67	0.87	44.44	40
	0.65	80	64.44	0.88	42.22	40
	0.66	77.78	60	0.89	42.22	40
	0.67	77.78	60	0.90	42.22	37.78
	0.68	75.56	60	0.91	40	37.78
	0.69	75.56	55.56	0.92	40	33.33
	0.70	73.33	53.33	0.93	40	33.33
	0.71	71.11	51.11	0.94	37.78	33.33
	0.72	66.67	48.89	0.95	37.78	33.33

www.admissions.ucla.edu/prospect/adm_fr/fracadrq.htm

admissions.berkeley.edu/sites/default/files/docs/Freshman_Flier.pdf

http://vi.wikipedia.org

http://vi.wiktionary.org