Dennis Diefenbach
email: dennis.diefenbach@univ-st-etienne.fr

Kamal Singh
email: kamal.singh@univ-st-etienne.fr

Pierre Maret
email: pierre.maret@univ-st-etienne.fr

WDAqua-core0: A Question Answering Component for the Research Community

Keywords: Question Answering, Qanary, QALD

We describe and present a new Question Answering (QA) component that can be easily used by the QA research community. It can be used to answer questions over DBpedia and Wikidata. The language support over DBpedia is restricted to English, while it can be used to answer questions in 4 different languages over Wikidata namely English, French, German and Italian. Moreover it supports both full natural language queries as well as keyword queries. We describe the interfaces to access and reuse it and the services it can be combined with. Moreover we show the evaluation results we achieved on the QALD-7 benchmark.

Introduction

Question answering (QA) is a very old research field in computer science. In the last two decades, thanks to the development of the Semantic Web, a lot of new structured data has become available on the web in the form of knowledge bases (KBs). Nowadays, there are KBs about media, publications, geography, life-science and more 1. The idea behind a QA system over KBs is to find the information, in a KB, requested by the user using natural language. This is generally addressed by translating a natural question to a SPARQL query that can be used to retrieve the desired information. We present here a QA component to answer questions over DBpedia and Wikidata that can answer both full and keyword natural language questions. It is integrated in the Qanary Ecosystem [START_REF] Diefenbach | The Qanary Ecosystem: getting new insights by composing Question Answering pipelines[END_REF] so that first, it can be easily reused by the research community and second, it takes advantage of the services available in Qanary.

Related work

In the context of QA, a large number of systems have been developed in the last years. For example, more than twenty QA systems were evaluated against the QALD benchmark2 . While many systems are querying DBpedia, we are only aware of one system querying wikidata, namely Platypus3

. Moreover most of the works address full natural language questions while only few address keyword questions. One exception is SINA [START_REF] Shekarpour | Sina: Semantic interpretation of user queries for question answering on interlinked data[END_REF].

The fact that QA systems often reuse existing techniques lead to the idea of developing QA systems in a modular way. Four frameworks tried to achieve this goal: QALL-ME [START_REF] Ferrández | The QALL-ME framework: A specifiable-domain multilingual Question Answering architecture[END_REF], openQA [START_REF] Marx | Towards an open question answering architecture[END_REF], the Open Knowledge Base and Question-Answering (OKBQA) challenge 4and Qanary [START_REF] Singh | Towards a message-driven vocabulary for promoting the interoperability of question answering systems[END_REF][START_REF] Both | Qanary a methodology for vocabulary-driven open question answering systems[END_REF][START_REF] Diefenbach | The Qanary Ecosystem: getting new insights by composing Question Answering pipelines[END_REF]. We integrated our QA component into the Qanary Ecosystem since it makes it easily reusable by the research community and offers a series of off-the-shelf services related to QA systems.

Description of WDAqua-core0

Our SPARQL creation algorithm uses a combinatorial approach based on the semantics encoded in the underlying KB. The full details will be disclosed in an upcoming publication as this is only a challenge submission. In the following we briefly describe the capabilities of WDAqua-core0. WDAqua-core0 can answer questions on both DBpedia and Wikidata. Note that the Wikidata dump5 contains binary and non-binary relationships. An example of a non-binary relationships expressing that the capital of Germany was Berlin from 1990 is expressed in two versions: @prefix wd: < http: // www . wikidata . org / entity / > . @prefix p: < http: // www . wikidata . org / prop / > . @prefix ps: < http: // www . wikidata . org / prop / statement / > . @prefix wdt: < http: // www . wikidata . org / prop / direct / > . @prefix pq: < http: // www . wikidata . org / prop / qualifier / > . @prefix wds: < http: // www . wikidata . org / entity / statement / > . wd:Q183 rdfs:label " Germany " @en ; wd:Q64 rdfs:label " Berlin " @en ; wdt:P36 rdfs:label " capital " @en ; # VERSION 1: reefied wd:Q183 p:P36 wds:q183 -7068 B86F . wds:q183 -7068 B86F a wikibase:Statement , ps:P36 wd:Q64 ; pq:P580 " 1990 -10 -03 T00:00:00Z " ^^xsd:dateTime .

# VERSION 2: non -reefied wd:Q183 wdt:P36 wd:Q64 ;

The first version uses properties with the namespaces p and ps while the second loses the temporal information and uses the namespace wdt. WDAqua-core0 is querying only the triples containing properties with namespace wdt. WDAqua-core0 can answer both keyword questions and questions in natural language. The complexity of the generated queries is limited to queries containing at most two triple patterns. The generated queries can be of type SELECT or ASK. The modifiers are limited to the COUNT operator. Thus, the questions with superlatives and comparatives can in general not be answered.

Finally it supports English on DBpedia and 4 different language over Wikidata, namely English, French, German and Italian. The evaluation is shown in section 5.

Integration in Qanary

Qanary is a framework to integrate QA components with the goal to make existing research in the QA field reusable. The QA component presented here is integrated into Qanary. A running version is registered into the Qanary service running under:

http://www.wdaqua.eu/qanary

In particular the component can be executed through RestFul interfaces. To run the service over a new question the RestFul interface under:

http://www.wdaqua.eu/qanary/startquestionansweringwithtextquestion can be used. Besides the generated answer, the top-30 generated queries can also be retrieved.

The integration into Qanary allows the combination of WDAqua-core0 with the other components and services that are already integrated into Qanary. In particular it can be combined with a speech recognition component and a language detection component. Additionally it can be used together with a number of services that are constructed around Qanary. These include a reusable front-end called Trill [START_REF] Diefenbach | Trill: A reusable front-end for qa systems[END_REF]. A demo of Trill that in the back-end uses WDAqua-core0 can be found under www.wdaqua.eu/qa. Figure 1 shows a screen-shot of Trill. Moreover WDAqua-core0 can be used together with some interfaces for user-feedback that are integrated into Trill [START_REF] Diefenbach | Introducing feedback in qanary: How users can interact with qa systems[END_REF]. One such feedback-interface can be seen in Figure 2. As a consequence WDAqua-core0 can be used by end-users and can for example be used to drive forward research in the domain of human-computer interaction. Finally Qanary has an interface that allows QA pipelines to be evaluated using Gerbil for QA 6. This means that WDAqua-core0 can be evaluated by the research community at all time especially when new benchmarks arise.

Evaluation over QALD-7

In this section we show the results of WDAqua-core0 over QALD-7 task 1 and task 4. We evaluate both over the keyword and the full-natural language questions. Moreover, we extended task 4 and introduced a new type of multilingual QA benchmark. QALD-7 task 1 requires to answer questions in multiple languages using data contained in the English DBpedia. In particular taking the Italian DBpedia to answer the Italian questions of QALD-7 task 1 does not work in general. The fact that the Italian questions must be answered using the English dataset, forces the systems to use translations. Instead we translate the questions of the QALD-7 task 4 into French, German and Italian and try to answer them using Wikidata. This is fundamentally different since in Wikidata the knowledge is the same and only the labels change. In particular a translation is not required, one can answer the Italian questions using an Italian dataset. The global (or macro) precision, recall and F-measure achieved over QALD-7 can be found in table 1. Note that WDAqua-core0 does not use a machine learning algorithm so there is not a problem of over-fitting the dataset. Fig. 2. Snapshot of the disambiguation interface for the question: "What is the capital of Germany?". By clicking on "Did you mean" several entities, the question might refereed to, are shown. These include the actual "Federal Republic of Germany" but also the "Capital of Germany" (as an entity), "West Germany", "East Germany", "Allied-Occupied Germany" and others. By clicking on the entity, the question is interpreted differently and a new answer is presented, e.g., if the user clicks on "West Germany", the answer "Bonn" is computed.

Conclusion

We have presented a QA component integrated into the Qanary Ecosystem that can be easily reused by the QA community. In particular it can used to push forward research in directions like the integration of speech recognition systems with QA systems and the interaction with users.

We have evaluated the component against QALD-7 in multiple aspects. We have shown the performance over both DBpedia and Wikidata with respect to keyword and full-natural language queries. Moreover, we have introduced a new type of multilingual QA benchmark that does not require translation but where the questions and the KB are in the same language. We have shown our results over this new type of multilingual QA benchmark.

Fig. 1 .

 1 Fig. 1. Screenshot of Trill using in the back-end WDAqua-core0 for the question "In which city is Jean Monnet University?"

Table 1 .

 1 The table shows the results of WDAqua-core0 over the QALD-7 training set.

	Task Dataset Language Type Precision Recall F-measure
	1 DBpedia	en	full	0.488 0.535	0.511
	1 DBpedia	en	keywords 0.374 0.407	0.390
	4 Wikidata	en	full	0.320 0.323	0.322
	4 Wikidata	en	keywords 0.280 0.280	0.280
	4 Wikidata	fr	full	0.123 0.131	0.127
	4 Wikidata	fr	keywords 0.326 0.365	0.344
	4 Wikidata	de	full	0.220 0.264	0.240
	4 Wikidata	de	keywords 0.301 0.349	0.325
	4 Wikidata	it	full	0.164 0.184	0.173
	4 Wikidata	it	keywords 0.188 0.207	0.197

http://lod-cloud.net

http://qald.sebastianwalter.org

https://askplatyp.us/?

http://www.okbqa.org/

https://dumps.wikimedia.org/wikidatawiki/entities/, https://www.mediawiki.org/wiki/ Wikibase/Indexing/RDF_Dump_Format

http://gerbil-qa.aksw.org

Acknowledgments Parts of this work received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skodowska-Curie grant agreement No. 642795, project: Answering Questions using Web Data (WDAqua).


Dennis Diefenbach
email: dennis.diefenbach@univ-st-etienne.fr

Youssef Dridi
email: youssef.dridi@etu.univ-st-etienne.fr

Kamal Singh
email: kamal.singh@univ-st-etienne.fr

Pierre Maret
email: pierre.maret@univ-st-etienne.fr

SPARQLtoUser: Did the question answering system understand me?

Keywords: Question answering system, User feedback, User interaction, SPARQL-ToUser, SPARQL2NL, Spartiqulator

In Question Answering (QA) systems, user questions are automatically converted into SPARQL queries. We present SPARQLtoUser, a simple method to produce a user understandable version of a SPARQL query so that the users can check what the QA system understood. Compared to existing similar methods, SPARQLtoUser is multilingual and is not restricted to one single KB, but easily portable to any KBs. This paper presents our method to generate the representation of SPARQL queries, and compares this method to existing ones. Moreover, we present the results of our user study that show that users have difficulties to understand whether the responses of a QA system are correct or not, with or without SPARQL representations.

Introduction

Question answering (QA) systems over Knowledge Bases (KB) are aiming to directly deliver an answer to a user's question. Achieving this with high precision is one important research area [START_REF] Diefenbach | Core Techniques of Question Answering Systems over Knowledge Bases: a Survey[END_REF]. Unfortunately the error rates are still very high. For example on popular benchmarks like WebQuestions [START_REF] Berant | Semantic Parsing on Freebase from Question-Answer Pairs[END_REF], SimpleQuestions [START_REF] Bordes | Large-scale simple question answering with memory networks[END_REF] and QALD 3 [9] F-measures of respectively 60 %, 70 %, 50 % are realistic. Also the precision is similar meaning that QA systems often do not retrieve the right answer for the user. Since the questions in the above mentioned benchmarks are generally all answerable over the given data, the F-measures in real scenarios are even lower. It is therefore important that users are able to understand if the information provided by a QA system is the one they asked for, so that they can easily estimate if they can believe the answers or not. In the following, we present a tool called SPARQLtoUser, which given a SPARQL query, produces a representation of it that can be shown to end-users. We compare our method with existing ones, namely SPARQL2NL [START_REF] Ngonga Ngomo | Sorry, i don't speak SPARQL: translating SPARQL queries into natural language[END_REF] and Spartiqulator [START_REF] Ell | Spartiqulation: Verbalizing sparql queries[END_REF]. Moreover, we show for the first time that users have difficulties to distinguish right answers from false ones.

Related work

QA over Knowledge Bases is a well-studied problem in computer science. For an overview of the published approaches we refer to [START_REF] Diefenbach | Core Techniques of Question Answering Systems over Knowledge Bases: a Survey[END_REF]. The task of presenting a SPARQL query to an end-user was tackled by verbalizing the query, i.e. by reformulating the SPARQL query using natural language. There are mainly two works that address this task: SPARQL2NL [START_REF] Ngonga Ngomo | Sorry, i don't speak SPARQL: translating SPARQL queries into natural language[END_REF] and Spartiqulation [START_REF] Ell | Spartiqulation: Verbalizing sparql queries[END_REF]. Both tools can be used to translate a SPARQL query to an English sentence. Both approaches were tested on DBpedia and MusicBrainz. Two main applications in the QA domain are used to motivate the need of such tools. The first was pointed out by Ell et al. [START_REF] Ell | Spartiqulation: Verbalizing sparql queries[END_REF]: "The verbalization allows the user to observe a potential discrepancy between an intended question and the system-generated query.". The second was pointed out by Ngonga Ngomo et al. [START_REF] Ngonga Ngomo | Sorry, i don't speak SPARQL: translating SPARQL queries into natural language[END_REF]. In this case, the idea is to verbalize all the SPARQL queries generated by a QA system and to let the user, if the answer is wrong, the possibility to choose a different SPARQL query by selecting the corresponding verbalization. We are not aware of any test of the efficacy of such approaches.

A simple representation of a SPARQL query for the user.

Users can ask questions in various languages, and they may target various KBs. Therefore, a generic method to build a representation of SPARQL queries (generated automatically from user questions) which is easily understandable by the users, should support multi-linguality and should be easily portable to different KBs. In this section, we present our method called SPARQLtoUser. We explain its origin and its mechanism.

The need for such a tool was derived from our QA webservice available under www.wdaqua.eu/qa. It uses in the backend a QA system called WDAquaCore0 [START_REF] Diefenbach | Wdaqua-core0: A question answering component for the research community[END_REF] which is integrated into the Qanary Ecosystem [START_REF] Diefenbach | The Qanary Ecosystem: getting new insights by composing Question Answering pipelines[END_REF] and is exposed to end-users through a reusable front-end called Trill [START_REF] Diefenbach | Trill: A reusable Front-End for QA systems[END_REF]. The QA system is multilingual, and we are able to port it easily to different KBs.

We observed that in many situations users would not be able to understand what was the interpretation of their questions by the QA system and, even as a developer, it was difficult to understand it by looking at the SPARQL query. Since the QA system supports different languages, and since different KBs can be queried, we came to develop a generic solution which is valid for any new languages and for new domains (i.e. KBs). In particular, the multilingual constraint brought us to not to try to verbalize the query, but to find a more schematic representation. An example can be seen in Figure 1. The missing verbalization can be seen as an additional burden for the user. One objective of this work is to measure and compare this burden to other approaches.

To generate the representation, we keep the original order of the triple patterns of the query and we substitute the URIs with the label in the corresponding language, i.e., the triple patterns

s 1,1 s 1,2 s 1,3 . . . s k,1 s k,1 s k,1
(where s indicates a slot) are converted to

f(s 1,1 ) slash(s 1,1 ) g(s 1,2 ) slash(s 1,3 ) f(s 1,3 ) . . . f(s k,1 ) slash(s k,1 ) g(s k,2 ) slash(s k,3 ) f(s k,3 ) where f(s i,j )=    label(s i,j ), for s i,j a URI "", for s i,j a variable s i,j , for s i,j a literal g(s i,j )=    label(s i,j ),
, for s i,j a URI {label(s)| (s a uri such that if s i,j is substituted with s the triple patterns has a solution in the KB )} , for s i,j a variable slash(s i,j )= / , for s i,j a URI "" , for s i,j a variable As a concrete example the query:

PREFIX wd : <http : / /www. w i k i d a t a . org/> SELECT DISTINCT ? x WHERE { wd : e n t i t y / Q79848 wd : prop / d i r e c t / P1082 ? x . } l i m i t 1000 is converted to: Southampton ( c i t y i n Hampshire , England ) / p o p u l a t i o n since "Q79848" refers to "Southampton" and "P1082" to the "population". We want to briefly describe the substitution of the predicates. If a predicate is an URI, we just put its label. If it is a variable, we expand it using all the properties that potentially could be placed in the variable position. This is motivated by the following example. In QA often hidden relations are expressed in the question. For example for the question "films by Charlie Chaplin" the proposition "by" can have different meanings like "film editor, screenwriter, director, composer, producer, cast member". WDAquaCore0 therefore produces as an interpretation: SELECT DISTINCT ? x WHERE { ? x ? p1 <http : / /www. w i k i d a t a . org / e n t i t y /Q11424> . ? x ? p2 <http : / /www. w i k i d a t a . org / e n t i t y /Q882> . } l i m i t 1000 where the properties are variables. SPARQLtoUser expands the variables ?p1 and ?p2 with the above described meanings. By showing this information, we want to show to the user what the QA system understood and give the users the lexicon to narrow down the search. Note that the label function can differ from the KB. Generally the strategy is to grab just the label, i.e. rdfs:label. But in some cases this does not identify the entity in an unambiguous way. For example in Wikidata the monument "Colosseum" and the band "Colosseum" have the same label. To identify the entity in a unique way, we add the description in parenthesis, i.e. "Colosseum (elliptical amphitheatre in Rome)" and "Colosseum (British band)". Depending on the KB a different property can be used to grab a verbalization of the resource and a property disambiguating it. Moreover for count queries, we surround the query with "How many" and for an ask query with "check". We limited the support of SPARQLtoUser to the types of queries that WDAquaCore0 can produce, but one can easily think about extension for this approach for other types of queries and aggregators. Note that the approach is easily portable to a new KB and that it is also multilingual in the sense that if the KB has not English labels the approach still can be applied. Table 1 gives an overview of the main differences between SPARLQ2NL, Spartiqulation and SPARQLtoUser.

The code of SPARQLtoUser can be found at https://github.com/WDAqua/ SPARQL_to_User. It is released under the MIT licence. The service is exposed as a webservice under the url https://wdaqua-sparqltouser.univ-st-etienne.fr/ sparqltouser. A description of the API can be found under https://github.com/ WDAqua/SPARQL_to_User/blob/master/README.md. As mentioned in the related works, one of the main motivations of SPARQL2NL and Spartiqulation was to allow users to better interact with QA systems. While both works performed a users study to test the quality of the verbalisation process, the impact of the SPARQL query representation was never analyzed in the context of QA.

We prepared a survey to test that up to which degree a SPARQL query representation helps the user to understand the interpretation of a user's question by a QA system.

To test the effectiveness of a SPARQL representation, we conducted a controlled experiment using A/B testing. In this approach, two versions of the same user-interface, with only one different UI-element, are presented to two different groups of users. The users are not aware that two different UI-elements are being tested. During A/B testing the objective is to measure whether the difference in the UI has an impact on some measurable variable, for example, the click-through rate of a banner advertisement.

Our objective was to test the impact of the SPARQL representation on the ability of a user, to understand the interpretation of a question by a QA system. We, therefore, choose 10 questions and presented to the users the answers that a QA system could have produced. Out of the 10 questions 4 were answered correctly while 6 were answered wrongly. Since our objective is to find out if users are able to determine whether the given answer is correct or not, we ask the users if they believe that the given answer is correct or not. Our objective in the A/B testing is to maximize the number of times the user believes the answer is right, when the answer is actually right, plus the number of times the user believes the answer is wrong, when it is actually wrong. We created 3 different versions of the same UI. The first, denoted A, does not contain any representation of the generated SPARQL query, the second, denoted B, contains the representation produced by SPARQLtoUser, and the third, denoted C, the one produced by SPARQL2NL. We did not test the interpretations given by Spartiqulation because the approach is similar to that of SPARQL2NL. An example of the 3 versions shown to the users can be seen in Figure 2. The list of all questions together with the answers can be found in Table 2. We tried to take questions that do not cover general knowledge so that it is highly probable that the users do not know the correct answer. The questions with incorrect answers generally present the problem of ambiguity. Sometimes the information provided with the answer, like the description or the top-k properties, allows to decide if the answer is correct or not. In other situations, the user can only choose to trust or not the given answer. The 3 forms send to the groups A, B, C can be found respectively under: A) https://goo.gl/forms/vCN7aZEaUAxWNMse2 B) https://goo.gl/forms/1fgauBGn5rb4JJ492 C) https://goo.gl/forms/WTibSJHjEBRgnaS62. The summary of the experimental results can be found in Table 2. In total 3 groups of 16 people answered our survey, i.e. for each version of the UI we have a sample size of 160 (16 people for 10 questions).

The experiments lead to the following conclusions. In general, users have big difficulties to understand the interpretation given by a QA system to the questions asked, independently of the approach. Without a representation of the SPARQL query the error rate is 34 %. With the representation given by SPARQLtoUser the error rate is 31 % and with the one of SPARQL2NL it is 28 %. Unfortunately, there is no significant difference between the different approaches. Also the statistical power is not high enough to say that there is no significant difference, i.e. we can not state with high enough probability that it makes no different to put or not a SPARQL representation. This is an unexpected result for us. We thought that the impact of the representation of the SPARQL query would be high enough to show a significant difference with a sample size of 160. The experiment shows that the impact is lower than what we expected. We want to discuss some unexpected observations. For example question 3 got better results without any SPARQL representation. Question 3 was "Who is the president of Eritrea ?". The presented answer is correct and the description of the answer says that clearly: "Isaias Afwerki Tigrinya is the first President of Eritrea, a position he has held since its independence in 1993.". In this case the representation of the SPARQL query given to the user seems to distract the user, since many users answer the question correctly without any interpretation. For the question "In which country was Claudia Cardinale born?" we assume that the QA system did a mistake when disambiguating the entity and returned the result for "Claudia Octavia" instead of "Claudia Cardinale". Both the representations of SPARQLtoUser and SPARQL2NL contain the string "Claudia Octavia" and not "Claudia Cardinale". Still many users believed that the answer is correct. Apparently, here the representation of the SPARQL query was ignored by many users.

Conclusion

Since current QA systems over KB still have low F-measures, it is important to let the users understand how the QA system interpreted their questions. We have presented a new representation of a SPARQL query for users. It is not based on verbalization, as done in previews works, but it is based on a simple rewriting of the query. The advantage is that this representation can be used with any language and it can be easily ported to new KBs. We have for the first time conducted a user study to measure the impact of SPARQL representations on the understand-ability of users of the interpretation of a question by a QA system. We found that if an impact exists then it is much smaller than we believed. More research in this area with quantitative methodologies is needed. In the future, we want to repeat our experiment with a larger number of participants and with more variations of the UI. In particular, we want to create a version that puts more emphasis on the SPARQL representation and one version that clearly states "I interpreted your question as: (followed by the interpretation)". The interaction of the users with a QA system is a poorly addressed, but interesting field.

Fig. 1 :

 1 Fig. 1: Snapshot of the Trill front-end. The interpretation generated by SPARQLtoUser is shown in blue over the answer "London").

Fig. 2 :

 2 Fig. 2: Three compared versions of the Trill front-end for the question "What is the capital of Eritrea?". A) with no SPARQL representation, B) with the representation of SPARQLtoUser, and C) with the representation of SAPRQL2NL.

Table 1 :

 1 Main features of SPARQLToUser, SPARQL2NL, Spartiqulator

	Tool	SPARQL support	Datasets (tested on)	Languages Portability
	SPARQL2NL part of SPARQL 1.1 syntax DBpedia, MusicBrainz	English	no
	Spartiqulator part of SPARQL 1.1	DBpedia, MusicBrainz	English	yes
	SPARQLToUser triple patterns, ASK and	DBpedia,	MusicBrainz,	multilingual	yes
		SELECT queries, COUNT	Wikidata, Dblp
		modifiers
	4 A user study to identify the efficacy of SPARQL
	representations for users.

Table 2 :

 2 This table shows the results of the user study. 10 questions different questions where shown to 48 people. These where divided into 3 equally sized groups and received 3 different versions of a UI following an A/B testing methodology. The three groups are denoted A,B,C. The UI shown to group A did not contain any representation of the SPARQL query, group B got the representation of SPARQLtoUser and group C the representation of SPARQL2NL.

http://www.sc.cit-ec.uni-bielefeld.de/qald/

Acknowledgments Parts of this work received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sk lodowska-Curie grant agreement No. 642795, project: Answering Questions using Web Data (WDAqua).


Dennis Diefenbach
email: dennis.diefenbach@univ-st-etienne.fr

Thomas Pellissier Tanon
email: thomas.tanon@ens-lyon.fr

Kamal Singh
email: kamal.singh@univ-st-etienne.fr

Pierre Maret
email: pierre.maret@univ-st-etienne.fr

Question Answering Benchmarks for Wikidata

Keywords: Wikidata, Question Answering Datasets, SimpleQuestions, WebQuestions, QALD, SimpleQuestionsWikidata, WDAquaCore0Questions

Wikidata is becoming an increasingly important knowledge base whose usage is spreading in the research community. However, most question answering systems evaluation datasets rely on Freebase or DBpedia. We present two new datasets in order to train and benchmark QA systems over Wikidata. The first is a translation of the popular SimpleQuestions dataset to Wikidata, the second is a dataset created by collecting user feedbacks.

Introduction

Wikidata is becoming an increasingly important knowledge base. Its popularity has grown after the termination of Freebase [START_REF] Bollacker | Freebase: a collaboratively created graph database for structuring human knowledge[END_REF] since August 31, 2016 and an effort has been made to migrate its content to Wikidata [START_REF] Pellissier Tanon | From Freebase to Wikidata: The great migration[END_REF]. The information in Wikidata is also partially overlapping with the one of DBpedia. One of the goals of Wikidata is to generate infobox of Wikipedia using the information in the Wikidata Knowledge Base. Since this is the main source of information of DBpedia the overlap will increase over time. Alltogether this means that Wikidata is becoming an important sources for general knowledge in the Semantic Web.

On the other side the most popular benchmarks for QA systems namely WebQuestions, SimpleQuestions and QALD are mainly considering Freebase and DBpedia as datasets.

We therefore see a strong need in datasets for evaluating QA systems over Wikidata. In the following, we present two new datasets for QA over Wikidata. The first is a translation of the SimpleQuestion dataset from Freebase to Wikidata, the second is a dataset that was created exploiting user feedback mechanism in a QA system.

Related work

The most popular benchmarks for QA over Knowledge Bases are WebQuestions [START_REF] Berant | Semantic Parsing on Freebase from Question-Answer Pairs[END_REF], SimpleQuestions [START_REF] Bordes | Large-scale simple question answering with memory networks[END_REF] and QALD 3 [8][5] [START_REF] Unger | Question answering over linked data (QALD-4)[END_REF][11] [START_REF] Unger | Cimiano: 6th Open Challenge on Question Answering over Linked Data (QALD-6)[END_REF]. Both WebQuestions and Simple-Questions were designed for Freebase. WebQuestions contains 5810 questions. They can be answered using one reified statement with potentially some constraints like type constraints or temporal constraints 4 . SimpleQuestions contains 108.442 questions which can be answered using one triple pattern.

Another popular benchmark is QALD. The number of questions and the datasets used in the different QALD challenges are reported in Table 1. They generally can be answered using up to 3 triple patterns. Sometimes modifiers like COUNT and aggregation operators are needed. Note that only in the last QALD challenge a benchmark for QA over Wikidata was presented. It contains 150 questions.

Some other less known benchmarks on top of Freebase exists, like Free917 [START_REF] Cai | Large-scale Semantic Parsing via Schema Matching and Lexicon Extension[END_REF] that provides 917 questions annotated with lambda calculus forms. In this section, we describe how we ported the SimpleQuestions dataset originally designed for Freebase to Wikidata. The SimpleQuestions dataset [START_REF] Bordes | Large-scale simple question answering with memory networks[END_REF] provides 108,442 questions, each annotated with a Freebase triple such that one of the acceptable answers to the question is the subject of the triple.

We mapped the Freebase triples to Wikidata using the same mapping process as [START_REF] Pellissier Tanon | From Freebase to Wikidata: The great migration[END_REF]: the subject and objects of triples, that are Freebase topics are mapped to Wikidata items using automatically generated mappings and the properties are mapped using a handmade mapping. When there is no equivalent property in Wikidata, but the Freebase inverse property has an equivalent property PXX in Wikidata, we map the Freebase property to a "fake" Wikidata property RXX ("R" indicating reverse). Note that not every translated triple is required to exist in Wikidata. When migrating the data from Freebase to Wikidata part of the information was lost. We therefore have created two versions, the first containing questions that can be answered in Wikidata, the second containing all questions. The QA community has concentrated so far on benchmarking QA systems assuming that most of the questions in the benchmark are answerable. But having many questions that are not answerable allows to tackle a new challenge, i.e. let the QA system decide if it has the knowledge to answer the question or not.

This new dataset contains 49.202 questions (21.957 of which are answerable over Wikidata). One of the reason of the gap in size between the two datasets is that we have only mapped 404 Freebase properties to Wikidata even if the dataset contains 1.837 properties. But the top 50 properties in the Freebase dataset provide 61% of the triples and the top 100, 76%. It allowed to map 45% of the dataset with 108 properties. The datasets is available under the Creative Commons Attribution 3.0 licence at https://github.com/askplatypus/wikidata-simplequestions. We offer it in the same format as the original SimpleQuestions dataset and in QALD format. We call it SimpleQuestionsWikidata.

Dataset using Logs and User feedback

Creating large datasets for QA is a tedious and expensive task. For example, [START_REF] Berant | Semantic Parsing on Freebase from Question-Answer Pairs[END_REF] report that they spent several thousands dollars for the creation of WebQuestions (containing 5810 questions) using Amazon Mechanical Turk. In the following, we show how it is possible to create a benchmark dataset for QA reducing the human effort and therefore also the cost. The idea is to involve users of a QA system in the creation of the benchmark. In this concrete case we collected the feedback given by users using WDAqua-core0 [START_REF] Diefenbach | Wdaqua-core0: A question answering component for the research community[END_REF], a QA system available under www.wdaqua.eu/qa. The web-service is online since June 2016 and received 12302 requests (5231 from them are unique). The QA service is exposed using Trill [START_REF] Diefenbach | Trill: A reusable front-end for qa systems[END_REF], a reusable front-end for QA systems. It contains a UI-component for feedback (see figure 1). The UI-component can be used mainly in two situations. In the first, end-users use the feedback component if they know that the answer is right or not. The second situation involves expert users that are able to understand SPARQL queries. If the answer is correct, the expert users can directly use the feedback component. If not they can check the top-k SPARQL queries generated by the QA system, as shown in Figure 2 and (if available) select the right one. The collected data contains 689 questions. Note that the questions are asked by real users and are a mixture between keyword and full natural language questions. The questions can be answered with maximal 2 triple patterns. Considering how the dataset was created the benchmark can contain errors. The resulting benchmark is available in QALD format at https://github.com/WDAqua/WDAquaCore0Questions under the Creative Commons Attribution 3.0 licence. We call the generated dataset WDAquaCore0Questions.

Conclusion

We have presented two new datasets for the research community. The first dataset is the translation of a popular benchmark over Freebase to Wikidata, namely Sim-pleQuestions which contains 21.957 questions answerable over Wikidata. The second is a dataset containing 689 questions generated using user feedback. By offering these datasets we hope to move the QA community towards Wikidata, which is becoming an increasingly important general Knowledge Base in the Semantic Web.

Fig. 1 :

 1 Fig. 1: Snapshot of the Trill front-end. In orange the UI-component for feedback is highlighted.

Fig. 2 :

 2 Fig. 2: Snapshot of the Trill front-end. The UI component allowing one to select between the generated SPARQL queries is opened. By clicking on the corresponding SPARQL query the corresponding result set is computed and shown to the user. Afterwards the UI component for feedback can be used.

Table 1 :

 1 Overview of the QALD benchmarks launched so far.

	Challenge Task	Dataset	Questions	Languages
	QALD-1	1 2	DBpedia 3.6 MusicBrainz	50 train / 50 test 50 train / 50 test	en en
	QALD-2	1 2	DBpedia 3.7 MusicBrainz	100 train / 100 test 100 train / 100 test	en en
	QALD-3	1 2	DBpedia 3.8 MusicBrainz	100 train / 99 test 100 train / 99 test	en, de, es, it, fr, nl en
		1	DBpedia 3.9	200 train / 50 test en, de, es, it, fr, nl, ro
	QALD-4	2 SIDER, Diseasome, Drugbank 25 train / 50 test	en
		3	DBpedia 3.9 with abstracts	25 train / 10 test	en
	QALD-5	1 2	DBpedia 2014 DBpedia 2014 with abstracts 50 train / 10 test 300 train / 50 test en, de, es, it, fr, nl, ro en
		1	DBpedia 2015	350 train / 100 test en, de, es, it, fr, nl, ro, fa
	QALD-6	2	DBpedia 2015 with abstracts 50 train / 25 test	en
		3	LinkedSpending	100 train / 50 test	en
		1	DBpedia 2016-04	214 train / 100 test en, de, es, it, fr, nl, hiIN
	QALD-7	2 DBpedia 2016-04 with abstracts 100 train / 50 test 3 DBpedia 2016-04 Syntetic questions	en en
		4	Wikidata	100 train / 50 test	en
	3 SimpleQuestions to Wikidata

http://www.sc.cit-ec.uni-bielefeld.de/qald/

https://www.microsoft.com/en-us/download/details.aspx?id=52763

Acknowledgments Parts of this work received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sk lodowska-Curie grant agreement No. 642795, project: Answering Questions using Web Data (WDAqua). It was also supported by the LABEX MILYON (ANR-10-LABX-0070) of Université de Lyon.