Hanna Suominen
email: hanna.suominen@anu.edu.au

Liadh Kelly
email: liadh.kelly@mu.ie

Lorraine Goeuriot
email: lorraine.goeuriot@imag.fr

The Scholarly Impact and Strategic Intent of CLEF eHealth Labs in 2012-2017

Keywords: Health Records, Information Extraction, Information Storage and Retrieval, Information Visualization, Speech Recognition

Since 2012, the CLEF eHealth initiative has aimed to gather researchers working on health text analytics and to provide them with annual shared tasks. This chapter reports on measuring its scholarly impact in 2012-2017 and describing its future objectives. The large number of submissions and citations demonstrate the substantial community interest in the tasks and their resources. Consequently, the initiative continues to run in 2018 and 2019 with its goal to support patients, their family, clinical staff, health scientists, and healthcare policy makers in accessing and authoring health information in a multilingual setting.

Introduction

The requirement to ensure that patients can understand their own privacy-sensitive, official health information in an Electronic Health Record (EHR) are stipulated by policies and laws. For example, the World Health Organization (WHO) 's Decla-ration on the Promotion of Patients' Rights in Europe from 1994 states that all patients in healthcare services have the right to be fully informed about their own health status, conditions, prognosis, diagnoses, discharge guidelines, and proposed and alternative treatment/non-treatment with risks, benefits, and progress. It also obligates healthcare workers to give each patient a written summary of this information and communicate in a way appropriate to this patient's capacity for understanding, including minimized use of unfamiliar jargon.

Improving the readability of EHRs can contribute to patients' partial control and mastery over their own health and care, leading to their increased independence from healthcare services, better health/care decisions, and decreased costs of care [START_REF] Mcallister | Patient empowerment: The need to consider it as a measurable patient-reported outcome for chronic conditions[END_REF]. This could mean replacing jargon words with patientfriendly synonyms, expanding shorthand, and an option to see the original text. The Systematized Nomenclature of Medicine -Clinical Terms (SNOMED CT) , Unified Medical Language System (UMLS) , and other terminology standards can help in defining synonym replacements and shorthand expansions, but Natural Language Processing (NLP) is needed to identify text snippets to be replaced with or extended by synonymous snippets. The enabling Information Extraction (IE) and NLP methods can also release healthcare workers' time from EHR-writing to, for example, longer discussions with the patient about the diagnosis, risks and benefits of the treatment options, and discharge guidelines.

Patient-friendly language in EHRs can help patients make informed decisions, but this also depends on their access to consumer leaflets and other supportive information about their health concerns in their personally-controlled EHR on the Internet. The large range of web content is widely accessible [START_REF] Ilic | The role of the internet on patient knowledge management, education, and decisionmaking[END_REF] and about 85 per cent of people turn to its search engines for health information [START_REF] Colineau | Talking about your health to strangers: understanding the use of online social networks by patients[END_REF]. EHRs can be used to naturally bridge patients' actions of reading their own EHR to searching supportive information; this Information Retrieval (IR) could mean enriching the EHR with hyperlinks to term definitions, care guidelines, and other information on patient-friendly and reliable sites on the Internet. Webbased EHRs that are targeted to both patients and healthcare workers for reading, writing, and sharing information are becoming increasingly common. 1Information access conferences have organized evaluation labs on related health NLP, IE, and IR tasks for almost 20 years, as illustrated below:

• The Text REtrieval Conference (TREC) has considered user profiling to filter in only the topically relevant biomedical abstract in its TREC Filtering Task in 2000 [START_REF] Robertson | The TREC-9 filtering track final report[END_REF]. Its TREC Genomics Tasks have ranged in 2003-2007 from ad-hoc IR to text classification, passage IR, and entitybased question answering on data from biomedical papers and de-identified EHRs [START_REF] Roberts | Tasks, topics and relevance judging for the TREC genomics track: Five years of experience evaluating biomedical text information retrieval systems[END_REF]. TREC Medical Records Task in 2011 targeted building a search engine where the patient cohort's eligibility criteria for a given study can be specified through the query and then after IR on de-identified EHRs, the matching population is returned for recruiting participants [START_REF] Voorhees | Overview of the TREC 2011 medical records track[END_REF]. [START_REF] Pestian | Sentiment analysis of suicide notes: A shared task[END_REF].

See [START_REF] Demner-Fushman | Aspiring to unintended consequences of natural language processing: A review of recent developments in clinical and consumer-generated text processing[END_REF] and [START_REF] Huang | Community challenges in biomedical text mining over 10 years: Success, failure and the future[END_REF] for recent reviews of evaluation labs and other developments in NLP, IE, and IR of healthcare worker and patient-authored EHRs. This chapter presents a review of CLEF eHealth outcomes in 2012-2017 and its strategic intent. The scholarly impact of the initiative is measured through the outcomes of problem specifications, resource releases, participation numbers, and citation counts. The paper extends [START_REF] Suominen | Scholarly influence of the Conference and Labs of the Evaluation Forum eHealth initiative: Review and bibliometric study of the 2012 to 2017 outcomes[END_REF] by providing further details about the annual citation analyses and widening the scope to the future.

Materials and Methods

Publication data from the CLEF proceedings relevant to CLEF eHealth 2012-2017 and papers that use the CLEF eHealth datasets, based on the reference catalog of the CLEF eHealth website, were reviewed.2 This literature review was supplemented by conducting a bibliometric study [START_REF] Tsikrika | The Scholarly Impact of CLEF[END_REF][START_REF] Angelini | Measuring and analyzing the scholarly impact of experimental evaluation initiatives[END_REF] of the reviewed publications and their citations received by 10 Nov 2017. Citation data for the publication data was collected from Google Scholar -one of the most comprehensive citation data sources [START_REF] Tsikrika | The Scholarly Impact of CLEF[END_REF][START_REF] Angelini | Measuring and analyzing the scholarly impact of experimental evaluation initiatives[END_REF]. In accordance with [START_REF] Tsikrika | The Scholarly Impact of CLEF[END_REF], we reviewed and refined the citation counts by hand for duplicated citation entries and incorrect entry merging.

Citation content analysis (Zhang et al, 2013a) was used for the publication and citation data analysis. This allowed testing of hypotheses about both the quantity and quality of the scholarly impact of CLEF eHealth in 2012-2017. See [START_REF] Suominen | Scholarly influence of the Conference and Labs of the Evaluation Forum eHealth initiative: Review and bibliometric study of the 2012 to 2017 outcomes[END_REF] for further methodological details.

Results

Placing layperson patients to the center of these shared tasks -opposed to clinical experts -as the targeted users is the main distinguishing feature of CLEF eHealth when comparing with earlier evaluation initiatives. In 2012, CLEF eHealth ran as a scientific workshop with an aim of establishing an evaluation campaign (Suominen (2012b[START_REF] Suominen | Text mining and information analysis of health documents[END_REF]), and from 2013-2017 this annual workshop has been supplemented with three or more shared tasks each year (Fig. 1). In 2013-2017, these tasks were patient-centric, with clinicians also considered from 2015, but in 2017 a pilot task on Technology Assisted Reviews (TAR) to support health scientists and policymakers' information access was also introduced [START_REF] Suominen | Overview of the ShARe/CLEF eHealth Evaluation Lab[END_REF][START_REF] Kelly | Overview of the ShARe/CLEF eHealth Evaluation Lab[END_REF][START_REF] Goeuriot | Overview of the CLEF eHealth Evaluation Lab[END_REF][START_REF] Kelly | Overview of the CLEF eHealth Evaluation Lab[END_REF][START_REF] Goeuriot | CLEF 2017 eHealth Evaluation Lab Overview[END_REF].

Problem Specifications

The The 2013 and 2014 Tasks 3, and 2014 Task 1 supplemented the processing of EHRs with information from the Internet, based on patient's information needs associated with the EHRs. The 2013 and 2014 Task 3 on information search considered English but in 2014 the problem was extended to serving an individual expressing their information need in a non-English language, for search on web-pages written in English because a large proportion of eHealth content on the Internet is written in English. The 2014 Task 1 on interactive IV had the overall goal of designing an effective, usable, and trustworthy web-environment for an English-speaking patient in their home in the USA to navigate, explore, and interpret health information as needed to promote understanding and informed decision-making.

In 2015 and 2016 the labs scope was expanded to multilingual text processing, medical web search, and speech-to-text conversion to ease both patients and clinicians' understanding of various types of medical content. The 2015 and 2016 Task 1 considered nursing handover report support in English. In clinical handover between nurses, verbal handover and note taking can lead to loss of information and electronic documentation is laborious, taking time away from patient education. The challenges addressed taking clinical notes automatically by using speech recognition to convert spoken nursing handover into digital text and IE to fill out a handover form, respectively. The 2015 and 2016 Task 2 considered clinical named entity recognition on French texts, which was previously an unexplored language. The challenges aimed to automatically identify clinically relevant entities from French biomedical articles. Also extracting causes of death from French death reports was considered. The 2015 and 2016 Task 3 considered patients' general information needs related to their medical complaints in a cross-lingual medical search on the web challenge. For example, their need to understand a condition or the cause of a medical symptom. The difficulty that this challenge focuses on is trying to extract relevant and reliable web pages that meet these needs expressed in English or several other languages.

The 2017 Task 1 continued the exploration of the problem of multilingual text processing, considering the IE of causes of death from both French and English death reports to ease clinicians' understanding. The 2017 Task 3 also continued its exploration of developing medical web search techniques to address the challenge posed by patients in locating relevant and reliable medical content. In addition the 2017 Task 2 considered a new challenge, that of technology assisted review generation in empirical medicine to support health care and policy making. Medical researchers and policy-makers while writing systematic review articles must ensure that they consider all documents relevant to their review. As the amount of medical literature continues to expand, automation in this process is necessary.

Data Releases

In 2013 Task 1, the de-identified, annotated EHRs were part of the Shared Annotated Resources (ShARe) corpus of the Multiparameter Intelligent Monitoring in Intensive Care (MIMIC) II database. 3 These 300 EHRs were authored in US intensive care. Each EHR was annotated by two people. A disorder name was defined as any text snippet which fulfills the following three conditions: 1) The snippet can be mapped to a concept in SNOMED CT. 2) This concept belongs to the semantic group of Disorder. 3) The concept belongs to one of the following semantic types in UMLS: Acquired abnormality, Anatomical abnormality, Cell or molecular dysfunction, Congenital Abnormality, Disease or Syndrome, Experimental Model of Disease, Injury or Poisoning, Mental or Behavioral Dysfunction, Neoplastic Process, Pathologic Function, or Signs and Symptoms. The same EHRs and annotations were used for 2013 Tasks 1b and supplemented by a new annotation for Task 2. Thirteen people were trained for the task and provided the visually annotated Task 1 EHRs. They were instructed to mark and, when possible, codify each clinical shorthand in the EHRs with one UMLS CUI or assign the code CUI-less.

An option to use the Task 1 and 2 EHRs and annotations for 2013 Task 3 was given but to enable IR, 55 new search tasks were formed (Goeuriot et al, 2013b). Two people formed the tasks from the Task 1 materials. For each search task, they generated a patient profile, information need, query title, and query description. The profile also allowed the participants to address the task without considering the EHRs. To create result document sets for these search tasks, about one million documents from commonly used health and medicine web-sites were used [START_REF] Hanbury | Khresmoi -multimodal multilingual medical information search[END_REF]. The relevance of each document was assessed by one person.

For the 2014 Task 1 on IV, six patient cases were chosen from these 2013 Tasks 1-3 data. 4The 2014 Task 2 on template filling used the original 300 EHRs from 2013 Task 1 and unseen 133 EHRs. The 2013 annotations were extended by focusing on the attributes-template filling for each disorder mention. Each EHR was annotated by two people. For 2014 Task 3, two people created 55 queries from the main disorders diagnosed in these EHRs. The 2013 document collection was used and associated result sets for the queries generated. The relevance of each document was assessed by one person. Participants were provided with the mapping between queries and EHRs, and were free to use the EHRs.

For 2015 Task 3 on IR, web-documents of the 2013 Task 3 were used. Queries were obtained by showing images and videos related to medical symptoms to users, who were then asked which queries they would issue to a web search engine if they were exhibiting such symptoms and thus wanted to find more information to understand these symptoms or which condition they were affected by. Twelve people generated the queries. A total of 266 unique queries were collected; of these, 67 queries in English were selected to be used in the task. The queries' translation was also provided into Arabic, Czech, German, Farsi, French, Italian, and Portuguese. Relevance and readability assessments were performed by four people.

The 2016 Task 3 on IR, used a new corpus, ClueWeb12 B13, 5 which is a large snapshot of the web (approx. 52.3 million web pages), crawled in Feb-May 2012. Unlike the dataset used in 2013-2015 IR Tasks, the corpus did not contain only health-related pages, making the dataset more in line with the material current web search engines index and retrieve. The queries extended upon the focus of the 2015 Task 3 (self-diagnosis) by considering real health information needs expressed by the general public through posts published in public health web forums. Forum posts were extracted from the 'askDocs' section of Reddit 6 and presented to six people, who were asked to formulate English queries based on what they read in the initial user post. This led to a set of query variants for a fixed number of topics. For the query variations element of the task, participants were told which queries relate to the same information need, to allow them to produce one set of results to be used as answer for all query variations of an information need. For the multilingual element of the task, Czech, French, German, Hungarian, Polish, and Swedish translations of the queries were provided. People assessed the outcomes for relevance, readability, and reliability. The 2017 Task 3 used the document collection and topics of 2016 Task 3, with the aim to acquire more relevance assessments and improve the collection re-usability.

The 2015 Task 1 and 2016 Task 1 on nursing handover report support used the NICTA Synthetic Nursing Handover Data (Suominen et al, 2015b). This set of 300 synthetic patient cases was developed for Speech Recognition (SR) and IE related to nursing shift-change handover in 2012-2015. Each case was authored by a a registered nurse and consisted of a patient profile; a written, free-form text paragraph to be used as a reference standard in SR; its spoken and speech-recognized counterparts; and human-annotations with respect to a form with 49 headings to fill out.

For 2015 Task 2 on IE, two types of biomedical documents were used: a total of 1, 668 titles of scientific articles indexed in the MEDLINE database, and six full text drug monographs published by European Medicines Agency (EMEA) . Annotations covered ten types of entities of clinical interest, defined by ten UMLS Semantic Groups. Three people marked each relevant entity mention in the documents, and assigned the corresponding semantic type(s) and CUIs [START_REF] Névéol | The QUAERO French medical corpus: A ressource for medical entity recognition and normalization[END_REF]. The new TAR in empirical medicine task -2017 Task 2 -used a subset of PubMed documents for its challenge to make Abstract and Title Screening more effective when judging whether to include/exclude a reference or considerer it for further examination at a full content level. The PubMed document IDs were collected by the task coordinators by re-running the MEDLINE Boolean query used in the original systematic reviews conducted by Cochrane to search fifty topics.

Software Releases

With an aim to lower the entry barrier and encourage novelty in problem solutions, CLEF eHealth began providing participants with software and code in 2013. These resource releases targeted method evaluation, EHR text annotation, and document relevance assessment, as illustrated below.

First, in 2013 Tasks 1 and 2, we released both a command-line tool and a graphical user interface that the participants could use to compute the values for the official and supplementary evaluation measures and visualize annotations against their method outputs. This extensible Human Oracle Suite of Tools7 [START_REF] South | Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text[END_REF] also supported them in annotating more data.

Second, in 2013 Task 3, we released the Relevation! tool8 [START_REF] Koopman | Relevation!: An open source system for information retrieval relevance assessment[END_REF]. We also provided a pointer to an established tool for computing values for the official and supplementary evaluation measures.

Third, 2016 Task 1 released the organizers' entire software stack as a state-of-theart solution to the handover problem (Suominen et al, 2015b).9 Participants were welcomed to use the released code for feature generation and/or IE, as intended, the results highlighted all participating teams' methods outperforming this known state-of-the-art baseline.

Papers and Their Citations

In 2012, the CLEF initiative introduced eHealth as a workshop that focused on eHealth documents and related analytics with a goal to spin out an evaluation lab. Its program consisted of three invited talks on collaborative datasets, resources, tools, and infrastructure; an expert panel; a student mentoring session where champions of the field provided feedback on designated PhD study plans and projects; a professional networking session; a working session for developing a road map for CLEF eHealth 2013; and oral talks for eleven papers (Table 1).

All CLEF eHealth 2012 talks focused on meeting the needs of healthcare professionals and patients in ease of information recording, access, and understanding via user-centered abbreviation processing, content generation, search engines, and vocabularies, among other tools to support patient-professional interaction across languages, sub-languages, and jargons. This community interest in the topic of userfriendly multilingual communication was verified in the workshop's roadmap session and formed the focus of the successful CLEF eHealth 2013-2017 labs (Figures 1 and2).

From 2012-2017, the 184 CLEF eHealth papers with 1, 299 citations generated in total the scholarly citation impact of almost 963, 000 (i.e., 741 × 1, 299) citations for the 741 co-authors and reached authors from 33 countries across the world (Tables 1-7). In accordance with the CLEF eHealth mission to foster teamwork, the number of co-authors per paper was 4 on average, with the maximum, median, minimum, and standard deviation of 15, 3, 1, and 3, respectively. In 47 out of the 184 papers (26%), this co-authoring collaboration was international. 143 out of the 184 papers (78%) had been cited at least once. The number of citations per paper varied from 0 to 147, with the mean and standard deviation of 7 and 15, respectively. The h-index and i10-index were 18 and 35, respectively. In 2012 CLEF eHealth resulted in 16 papers and in 2013-2017, this number increased to 31-35.

Discussion

CLEF eHealth has been contributing to evaluation initiatives in medical NLP and IR since 2012. Evaluation resources have been developed and shared with the community to support the understanding of and access to health content by laypeople, clinicians, and policy-makers. In so doing the lab has provided an evaluation setting for the progression of multilingual eHealth Information and Communications Technology (ICT) research and development. The annual eHealth workshop held at the main CLEF conference provides for the dissemination and discussion of the outcomes of each year's tasks. Each year the organizers produce overview papers describing the tasks offered and participants results. These have proven influential, as indicated by their citation indexes.

Although the CLEF eHealth installations have attracted substantial community interest, as reflected by the 741 co-authors of the 184 papers from 33 countries, we have not attracted participation from Central America. Also substantially more participation from Africa, South America, and the Middle East should be achievable. However, this problem of insufficient participation has been acknowledged in a review of biomedical evaluation initiatives by [START_REF] Huang | Community challenges in biomedical text mining over 10 years: Success, failure and the future[END_REF] as one of their main conclusions.

By virtue of the lab series over the first seven years of it's life, from 2012-2018 inclusive, providing access to shared data, resources, processing methods, and evaluation settings for medical system research, development and evaluation; offering reproducibility, scalability, and user-centricity; and finally bringing the research community together through the lab series to collaborate and discuss challenges associated with technique development in medical NLP and IR, we conjecture that CLEF eHealth has impacted progress in these spaces. While it is difficult to ac- curately quantify such impact, the 1, 299 citations, with impact of circa 963, 000 generated by the lab in it's first six year's of existence are suggestive. Progress in the areas addressed by the lab has the potential to generate high impact not only on the research field, but more generally on society, given the importance of health information access to support healthcare as well as to empower people to manage their health. Consequently, CLEF eHealth runs in 2018 and 2019 extending its previous challenges.

Going forward, the strategic intent of the CLEF eHealth initiative is to develop shared tasks that impact patient care by impacting (1) patient understanding of their health and healthcare, and (2) the entire healthcare ecosystem which exists to support patient care. To achieve this, we continue to provide the community with an increasingly sophisticated dataset of clinical narrative, enriched with links to evidence-based care guidelines, systematic reviews, and other further information, to advance the state-of-the-art in multilingual NLP and IR in healthcare. Our scope fosters student mentoring, diverse collaboration, and reproducible research by welcoming and supporting new participants; facilitating multi-professional and interdisciplinary collaboration; and encouraging participants to reflect on methods and practical steps to take to facilitate the replication of their experiments; fostering the release of open-source datasets and tools to reach a wider community. This scope is supported by an increasing interest of the community in health-related IR and NLP, and its increased consideration for shared tasks.

Fig. 1

 1 Fig. 1 Timeline of CLEF eHealth

  The 2016 Task 2 extended this 2015 Task 2 data release by including 833 MEDLINE titles and 4 EMEA documents, with annotations for ten types of clinical entities with UMLS normalization. In another challenge, it used 65, 843 death certificates from the CépiDC Causes of Death Corpus that were manually coded with International Classification of Diseases (ICD) -10, as per the WHO standards. The 2017 Task 1 supplemented these French death certificates by those in English from the USA. The annotators at the French National Institute of Health and Medical Research (INSERM) in 2006-2013 and the US Center for Disease Control in 2015 also manually built dictionaries of terms associated with the codes. Several versions of these lexical resources were supplied to participants.

Fig. 2

 2 Fig. 2 Participation in the CLEF eHealth evaluation labs

•

  Prior to Conference and Labs of the Evaluation Forum (CLEF) introducing its Electronic Health (eHealth) initiative in 2012, its ImageCLEF initiative organized annual ImageCLEFmed tasks since 2005 on image annotation, image search, and automated form filling related to image analysis tasks (Kalpathy-Cramer et al, 2011). • In parallel in 2005-2012, the Informatics for Integrating Biology and the Bedside initiative has also been addressing eHealth NLP through its following shared tasks (Uzuner et al, 2011; Sun et al, 2013): text de-identification and identification of smoking status in 2006; recognition of obesity and comorbidities in 2008; medication IE in 2009; concept, assertion, and relation recognition in 2010; coreference analysis in 2011; and temporal relations challenge in 2012. • Also the Medical NLP Challenges have targeted automated diagnosis coding of radiology reports in 2007 and classifying the emotions found in suicide notes in 2011

Task Timeline 2012 2013 2014 2015 2016 2017 Information Extraction

		Named entity	Electronic	Biomedical articles in French
		recognition and/or	heath
		normalisation	records
			(EHRs) in
			English
		Extraction		EHRs in			Multi-lingual
				English			death reports
		Classification				Death reports
						in French
		Replication				Code	Code
		Visualisation		EHRs and
				other
				electronic
				health
	Information			(eHealth) data
	Management			in English
		Report generation			Nursing handover reports in
		and management			English
		Patient-centered	Multilingual eHealth data
		information retrieval
	Retrieval Information	information retrieval Cross-lingual		Multilingual eHealth data
		Technology assisted						Bio-medical
		reviews in empirical						articles in
		medicine						English
		Yes	Yes	Yes	Yes	Yes		Yes
	Workshop
		No. of expressions of		175	220	90	116	117
		interests to participate
		No. of participating		34	24	20	20	34
		teams
		No. of papers in the	16	34	29	24	24	35
		CLEF proceedings
	Participation	No. of authors	50	162	107	91	113	128
		No. of authors'	35	85	69	50	73	82
		affiliations	(32, 2, 1)	(74, 4, 7)	(65, 0, 4)	(47, 2, 1)	(64, 6, 3)	(75, 5, 2)
		(academia, industry,
		government)
		No. of affiliated	8	10	22	19	16	22
		countries

Table 1

 1 Bibliometric analysis of the CLEF eHealth 2012 lab workshop on 26 Oct 2017

	ID Paper	Authors Authors' countries	Citations
	Organizers' overview:
	1	Suominen (2012a)	1	Australia	2
	Participants' papers:
	2	Friberg Heppin and Järvelin	2	Sweden	1
		(2012)
	3	Heimonen et al (2012)	3	Finland	0
	4	N. Isenius (2012)	3	Sweden	10
	5	Kanhov et al (2012)	3	Sweden	0
	6	Kelly et al (2012)	4	Austria, Ireland	1
	7	Kreiner et al (2012)	4	Austria	0
	8	Laippala et al (2012)	5	Finland	0
	9	Martinez et al (2012)	4	Australia	5
	10 Moen and Marsi (2012)	2	Norway	0
	11 Suominen et al (2012a)	8	Australia	1
	12 Suominen et al (2012b)	3	Australia, Austria	0
	Invited abstracts:
	13 Chapman (2012)	1	USA	0
	14 Hanlen (2012)	1	Australia	0
	15 Jones et al (2012)	5	Austria, Finland, Ireland, Sweden,	0
				Switzerland
	16 Suominen (2012b)	1	Australia	0

Table 2

 2 Bibliometric analysis of the CLEF eHealth 2013 evaluation lab on 31 Oct 2017

	ID Paper	Authors Authors' countries	Citations
	Organizers' overviews:
	1	Suominen et al (2013)	15	Australia, Finland, Ireland, Sweden,
				USA
	2	Pradhan et al (2013)	9	Australia, USA
	3	Mowery et al (2013)	11	Australia, Finland, USA
	4	Goeuriot et al (2013a)	9	Australia, Austria, Finland, Ireland,
				Switzerland
	Participants' papers for 2013 Task 1):
	5	Bodnari et al (2013)	5	France, USA
	6	Cogley et al (2013)	3	Ireland
	7	Fan et al (2013)	3	USA
	8	Gung (2013)	1	USA
	9	Hervas et al (2013a)	4	Spain
	10 Hervas et al (2013b)	4	Spain
	11 Leaman et al (2013)	3	USA
	12 Liu et al (2013)	4	USA
	13 Osborne et al (2013)	3	USA
	14 Patrick et al (2013a)	3	Australia
	15 Ramanan et al (2013)	3	India
	16 Tang et al (2013)	5	China, USA
	17 Wang and Akella (2013)	2	USA
	18 Xia et al (2013a)	7	China
	19 Zuccon et al (2013a)	4	Australia
	Participants' papers for 2013 Task 2:
	20 Jagannathan et al (2013)	7	USA
	21 Patrick et al (2013b)	3	Australia
	22 Wu et al (2013)	6	USA
	23 Xia et al (2013b)	7	China
	24 Zweigenbaum et al (2013)	5	France
	Participants' papers for 2013 Task 3:
	25 Bedrick and Sheikshabbafghi	2	USA
		(2013)
	26 Caballero Barajas and Akella	2	USA
		(2013)
	27 Chappell and Geva (2013)	2	Australia
	28 Choi and Choi (2013)	2	Republic of Korea
	29 Limsopatham et al (2013)	3	UK
	30 Zhang et al (2013b)	5	USA
	31 Zhong et al (2013)	6	China
	32 Zhu et al (2013)	5	USA
	33 Zuccon et al (2013b)	3	Australia
	Participants' papers for student mentoring:
	34 Murtola et al (2013)	6	Finland, India, Norway

Table 3

 3 Bibliometric analysis of the CLEF eHealth 2014 evaluation lab on 9 Nov

	ID Paper	Authors Authors' countries	Citations
	Organizers' overviews:
	1	Kelly et al (2014)	11	Australia, Austria, Germany, Ireland,
				Sweden, USA
	2	Suominen et al (2014)	10	Australia, Germany, Ireland, USA
	3	Mowery et al (2014)	11	Australia, Ireland, Sweden, USA
	4	Goeuriot et al (2014b)	9	Australia, Austria, Czech Republic,
				Ireland, Switzerland
	Participants' papers for 2014 Task 1:
	5	Hyman and Fridy (2014)	2	USA
	Participants' papers for 2014 Task 2:
	6	Hamon et al (2014)	3	France
	7	Herbst et al (2014)	4	Germany
	8	Huynh and Ho (2014)	2	Vietnam
	9	Johri et al (2014)	3	India, Japan
	10 Liu and Ku (2014)	2	Taiwan
	11 Liu et al (2014)	4	Canada
	12 Mkrtchyan and Sonntag (2014) 2	Germany
	13 Osborne (2014)	1	USA
	14 Ramanan and Senthil Nathan	2	India
		(2014)
	15 Sequeira et al (2014)	4	Portugal
	Participants' papers for 2014 Task 3:
	16 Choi and Choi (2014)	2	Republic of Korea
	17 Claveau et al (2014)	4	France
	18 Dramé et al (2014)	3	France
	19 Malagon and L/'opez (2014)	2	Spain
	20 Nesrine et al (2014)	3	Tunisia
	21 Oh and Jung (2014)	2	Republic of Korea
	22 Ozturkmenoglu et al (2014)	3	Turkey
	23 Saleh and Pecina (2014)	2	Czech Republic
	24 Shenwei et al (2014)	4	Canada
	25 Thakkar et al (2014)	4	India
	26 Thesprasith and Jaruskulchai	2	Thailand
		(2014)
	27 Verberne (2014)	1	Netherlands
	28 Wu and Huang (2014)	2	Canada
	29 Yang et al (2014)	3	USA

Table 4

 4 Bibliometric analysis of the CLEF eHealth 2015 evaluation lab on 10 Nov 2017

	ID Paper	Authors Authors' countries	Citations
	Organizers' overviews:
	1	Goeuriot et al (2015)	8	Australia, Austria, France, Ireland
	2	Suominen et al (2015a)	5	Australia, France, Ireland
	3	Névéol et al (2015)	7	France
	4	Palotti et al (2015a)	8	Australia, Austria, Czech Republic,
				France, Ireland
	Participants' papers for 2015 Task 1:
	5	Herms et al (2015)	4	Germany
	6	Luu et al (2015)	4	Australia
	Participants' papers for 2015 Task 2:
	7	Afzal et al (2015)	5	Netherlands
	8	Chernyshevich and Stankevitch	2	Belarus
		(2015)
	9	Cotik et al (2015)	3	Argentina, Spain
	10 D'Hondt et al (2015b)	6	France
	11 Jain (2015)	1	India
	12 Jiang et al (2015)	3	China
	13 Soualmia et al (2015)	4	France
	Participants' papers for 2015 Task 3:
	13 As above
	14 D'Hondt et al (2015a)	3	France
	15 Ghoddousi and Huang (2015)	2	Canada
	16 Huynh et al (2015)	3	Vietnam
	17 Ksentini et al (2015)	4	France, Tunisia
	18 Liu and Nie (2015)	2	Canada
	19 Lu (2015)	1	China
	20 Oh et al (2015)	3	Republic of Korea
	21 Saleh et al (2015)	3	Czech Republic
	22 Song et al (2015)	5	China, USA
	23 Thesprasith and Jaruskulchai	2	Thailand
		(2015)
	24 Thuma et al (2015)	3	Botswana

Table 5

 5 Bibliometric analysis of the CLEF eHealth 2016 evaluation lab on 10 Nov 2017 National Information and Communications Technologies Australia (NICTA) , funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program; Protégé resource, supported by grant GM10331601 from the National Institute of General Medical Sciences of the United States (of America) (US) National Institutes of Health; PhysioNetWorks Workspaces; ShARe project funded by the US National Institutes of Health (R01GM090187); Swedish Research Council (350-2012-6658); Swedish Vårdal Foundation; US Department of Veterans Affairs (VA) Consortium for Healthcare Informatics Research (CHIR) ; and US Office of the National Coordinator of Healthcare Technology, Strategic Health Information Technology Advanced Research Projects (SHARP) 90TR0002.

	ID Paper	Authors Authors' countries	Citations
	Organizers' overviews:
	1	Kelly et al (2016)	6	Australia, Austria, France, Ireland
	2	Suominen et al (2016)	4	Australia, France, Ireland
	3	Névéol et al (2016)	11	France, Ireland
	4	Zuccon et al (2016a)	9	Australia, Austria, Czech Republic,
				France, Ireland, Switzerland
	Participants' papers for 2016 Task 1:
	5	Ebersbach et al (2016)	4	Germany
	6	Quiroz et al (2016)	4	Netherlands
	7	Song et al (2016a)	6	China
	Participants' papers for 2016 Task 2:
	8	Cabot et al (2016)	4	France
	9	Dermouche et al (2016)	6	France
	10 Ho-Dac et al (2016)	8	France
	11 Mottin et al (2016)	6	Switzerland
	12 Saleh and Pecina (2016)	2	Czech Republic
	13 van Mulligen et al (2016)	5	Netherlands
	14 Vivaldi et al (2016)	3	Argentina, Spain
	15 Zweigenbaum and Lavergne	8	France
		(2016)
	Participants' papers for 2016 Task 3:
	16 Budaher et al (2016)	3	France
	17 Oh and Jung (2016)	2	Republic of Korea
	18 Silva and Lopes (2016)	2	Portugal
	19 Soldaini et al (2016)	3	USA
	20 Song et al (2016b)	6	China
	21 Thuma et al (2016)	3	Botswana
	22 Ullah and Aono (2016)	2	Japan
	23 Wang et al (2016a)	3	China
	24 Wang et al (2016b)	3	USA
	257528; Microsoft Azure for Research Award CRM:0518649; MIMIC II Database;

Table 6

 6 Bibliometric analysis of the CLEF eHealth 2017 evaluation lab on 10 Nov 2017

	ID Paper	Authors Authors' countries	Citations
	Organizers' overviews:
	1	Goeuriot et al (2017)	9	Australia, Austria, France, Ireland,
				Netherlands
	2	Névéol et al (2017)	9	France, USA
	3	Kanoulas et al (2017)	4	Netherlands, UK
	4	Palotti et al (2017)	8	Australia, Austria, Czech Republic,
				France, Ireland
	Participants' papers for 2017 Task 1:
	5	Atemezing (2017)	1	France
	6	Cabot et al (2017)	3	France
	7	Di Nunzio et al (2017b)	4	Italy
	8	Ebersbach et al (2017)	3	Germany
	9	Ho-Dac et al (2017)	12	France
	10 Jonnagaddala and Hu (2017)	2	Australia, Ireland
	11 Miftahutdinov and Tutubalina	2	Russia
		(2017)
	12 Seva et al (2017)	4	Germany
	13 Tchechmedjiev et al (2017)	4	France, USA
	14 Zweigenbaum and Lavergne	2	France
		(2017)
	Participants' papers for 2017 Task 2:
	15 Alharbi and Stevenson (2017)	2	UK
	16 Anagnostou et al (2017)	4	Greece
	17 Azzopardi et al (2017)	3	UK
	18 Chen et al (2017)	7	China
	19 Cormack and Grossman (2017) 2	Canada
	20 Di Nunzio et al (2017a)	4	Italy
	21 Lee (2017)	1	Singapore
	22 Norman et al (2017)	3	France, Netherlands
	23 Scells et al (2017)	4	Australia
	24 Singh and Thomas (2017)	2	India
	25 Singh et al (2017)	4	UK, USA
	26 van Altena and Delgado Olabar-	2	Netherlands
		riaga (2017)
	27 Yu and Menzies (2017)	2	USA
	Participants' papers for 2017 Task 3:
	28 Diaz-Galiano et al (2017)	5	Spain
	29 Hollmann and Eickhoff (2017) 2	Switzerland
	30 Jimmy et al (2017)	3	Australia, Indonesia
	31 Oh and Jung (2017)	2	Republic of Korea
	32 Palotti and Rekabsaz (2017)	2	Austria
	33 Saleh and Pecina (2017)	2	Czech Republic
	34 Thuma et al (2017)	3	Botswana
	35 Yang and Goncalves (2017)	2	Portugal

Table 7

 7 Bibliometric analysis of other papers that use CLEF eHealth data on 10 Nov 2017

	ID Paper			Authors Authors' countries	Citations
	1	Goeuriot et al (2013b)		8	Australia, Austria, Ireland, Switzer-	11
						land
	2	Goeuriot et al (2014a)		3	Ireland	6
	3	Kholghi et al (2014)		4	Australia	5
	4	Pradhan et al (2014)		9	Australia, USA	53
	5	Suominen (2014)		1	Australia, Finland	4
	6	Zuccon and Koopman (2014)	2	Australia	22
	7	De Vine et al (2015)		4	Australia	5
	8	Kholghi et al (2015)		4	Australia	5
	9	Palotti et al (2015b)		3	Australia, Austria	6
	10 Suominen et al (2015b)		4	Australia, Finland	15
	11 Zhou and Suominen (2015)	2	Australia, Finland	0
	12 Zhou et al (2015)		3	Australia, Finland	2
	13 Zuccon et al (2015)		3	Australia, Austria	21
	14 Beloborodov	and	Goeuriot	2	France, Russia	6
		(2016)
	15 Goeuriot et al (2016)		4	Australia, Austria, France, Ireland	2
	16 Kholghi et al (2016)		4	Australia	7
	17 Mowery et al (2016)		13	Australia, Finland, Sweden, USA	1
	18 Palotti et al (2016a)		4	Australia, Austria, France	6
	19 Palotti et al (2016b)		5	Australia, Austria, France	21
	20 Rekabsaz et al (2016)		4	Australia, Austria	7
	21 Zuccon (2016)			1	Australia	17
	22 Zuccon et al (2016b)		3	Australia, Austria	8

They have been open, for example, in Estonia (http://www.e-tervis.ee) and Australia (https://myhealthrecord.gov.au) since

and 2012, respectively.

https://sites.google.com/site/clefehealth

https://www.clinicalnlpannotation.org, http://mimic.physionet.org

After the task, the workspace was kept open for registration at https://physionet.org/ works/CLEFeHealth2014Task1/; by 26 Oct 2017, access had been granted to 60 people.

http://lemurproject.org/clueweb12/index.php

https://www.reddit.com/r/AskDocs/

http://blulab.chpc.utah.edu/content/ehost-extensible-humanoracle-suite-tools

8 http://ielab.github.io/

relevation 9 https://www.kaggle.com/c/hospital-handover-forms/

Acknowledgements We gratefully acknowledge the people involved in CLEF eHealth 2012-2017 as participants or organizers. We are also thankful to support by the French National Research Agency (ANR) , under grant CABeRneT ANR-13-JS02-0009-01; CLEF Initiative; Data61; European Science Foundation (ESF) Project EvaLuating Information Access Systems (ELIAS) network program Horizon 2020 program (H2020-ICT-2014-1) under grant agreement 644753 (KConnect); Knowledge Helper for Medical and Other Information users (Khresmoi) project, funded by the European Union Seventh Framework Programme (FP7) /2007-2013 under grant agreement No.