MSc Ivan Lerner

MSc Nicolas Paris

PhD Xavier Tannier
email: xavier.tannier@sorbonne-universite.fr

Terminologies augmented recurrent neural network model for clinical named entity recognition

Keywords: "clinical natural language processing", "named entity recognition", "information extraction", "machine learning"

Objective

We aimed to enhance the performance of a supervised model for clinical named-entity recognition (NER) using medical terminologies. In order to evaluate our system in French, we built a corpus for 5 types of clinical entities.

Methods

We used a terminology-based system as baseline, built upon UMLS and SNOMED. Then, we evaluated a biGRU-CRF, and a hybrid system using the prediction of the terminologybased system as feature for the biGRU-CRF. In French, we built APcNER, a corpus of 147 documents annotated for 5 entities (Drug names, Signs or symptoms, Diseases or disorders, Diagnostic procedures or lab tests and Therapeutic procedures). We evaluated each NER systems using exact and partial match definition of F-measure for NER. The APcNER contains 4,837 entities, which took 28 hours to annotate. The inter-annotator agreement as measured by Cohen's Kappa was substantial for non-exact match ( = 0.61) and moderate considering exact match ( = 0.42). In English, we evaluated the NER systems on the i2b2-2009 Medication Challenge for Drug name recognition, which contained 8,573 entities for 268 documents, and i2b2-small a version reduced to match APcNER number of entities.

Results

For drug name recognition on both i2b2-2009 and APcNER, the biGRU-CRF performed better that the terminology-based system, with an exact-match F-measure of 91.1% versus 73% and 81.9% versus 75% respectively. For i2b2-small and APcNER, the hybrid system outperformed the biGRU-CRF, with an exact-match F-measure of 85.6% versus 87.8% and 88.4% versus 81.9% respectively. On APcNER corpus, the micro-average F-measure of the hybrid system on the 5 entities was 69.5% in exact match and 84.1% in non-exact match.

Conclusion

APcNER is a French corpus for clinical-NER of five types of entities which covers a large variety of document types. The extension of the supervised model with terminology has allowed an easy increase in performance, especially for rare entities, and established near state of the art results on the i2b2-2009 corpus.

Introduction

Within the range of data covered by electronic health records (EHRs), clinical documents (e.g., discharge summaries or physicians' letters) are rich sources of information for various applications such as patient enrollment in clinical research studies, epidemiological surveillance, medical coding and decision-making tools [START_REF] Wang | Clinical information extraction applications: A literature review[END_REF]. Information extraction tools must be tailored for applications in the medical field, where the language is both unstructured (e.g., free text) and semi-structured (e.g., drug lists), with a wide vocabulary.

Named-entity recognition (NER) is the process of mention detection and type classification of named entities, where named entities are concepts that can be referenced by various linguistic expressions. In recent decades, there has been a growing interest in clinical-NER, the task of NER for medical concepts such as drug names, diseases, or signs [START_REF] Liu | Learning for Biomedical Information Extraction: Methodological Review of Recent Advances[END_REF].

Supervised systems based on machine learning have proven to be more efficient than rulebased and terminology-based systems for NER [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF]. Research efforts have then been made to unify these methods in hybrid systems, in a purely unsupervised [START_REF] Zhang | Unsupervised biomedical named entity recognition: experiments with clinical and biological texts[END_REF][START_REF] Alicante | Unsupervised entity and relation extraction from clinical records in Italian[END_REF] or semi-supervised fashion [START_REF] Vigna | Semi-supervised Knowledge Extraction for Detection of Drugs and Their Effects[END_REF][START_REF] Gupta | Semi-Supervised Recurrent Neural Network for Adverse Drug Reaction mention extraction[END_REF][START_REF] Fries | SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Data[END_REF]. Such approaches are motivated by the necessity to reduce the need for manually annotated examples in the case of a supervised system, or the need for handwritten rules by experts in the field of rule-based and ontological systems. For instance, a supervised model can automatically learn context cues for the various expressions that are associated with the mentions of drug names, such as "patient was given […]" or "[...] was stopped early." A model relying on such context cues would increase its ability to detect new drug names or variants of drug name expressions that are not yet included in terminologies.

In addition, for languages other than English, annotated corpora and ontologies are scarcer.

For instance, in French, there is only one annotated clinical corpus which covers a small subset of the medical domain [START_REF] Campillos | A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)[END_REF], and international ontologies such as the Unified Medical Language System (UMLS®) are not fully translated [START_REF] Névéol | Language Resources for French in the Biomedical Domain[END_REF]. The development cost of such annotated corpus is very high, as it has been reported that annotating 5 medical documents for 12 entities take on average 82 mins [START_REF] Campillos | A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)[END_REF].

In this study, we aimed to evaluate a clinical-NER system for which development could scale up to the different uses of a large French data warehouse [START_REF] Daniel | Initializing a hospital-wide data quality program. The AP-HP experience[END_REF]. First, we built an annotated corpus for five clinical entity types, for which we present here the details of the annotation process. Second, we evaluated three different systems: 1) a terminology-based system built upon the Unstructured Information Management Architecture (apache-UIMA®) framework, 2) a supervised neural model based on a biGRU-CRF architecture, and 3) a hybrid system. This comparison focused on drug name recognition and was achieved on our French corpus as well as on a well-known, freely available corpus in English, for comparison purposes. On the French corpus exclusively, we evaluated the three systems on the task of jointly detecting the 5 types of annotated entities (Drug names, Signs or symptoms, Diseases or disorders, Diagnostic procedures or lab tests and Therapeutic procedures).

Methods

Corpus and annotation process

We used two corpora in this study, an English corpus from the i2b2-2009 Medication challenge (i2b2-2009) [START_REF] Uzuner | Community annotation experiment for ground truth generation for the i2b2 medication challenge[END_REF] and a French corpus APcNER, with clinical reports extracted from the Assistance Publique -Hôpitaux de Paris (AP-HP) data warehouse [START_REF] Daniel | Initializing a hospital-wide data quality program. The AP-HP experience[END_REF].

i2b2-2009 corpus

The original corpus included 1243 de-identified discharge summaries, 17 of which were annotated by the i2b2 team, and 251 collectively annotated by the challenge participants. 1Overall, the 268 annotated documents of the corpus contained 337,745 tokens, 8,573 entities and 17,933 sentences, for a vocabulary of 23,214 tokens. The median sentence length was 13. The overall 8,573 entities comprised 6,488 (75%) unigrams, 1,053 (12%) bigrams and 1,029 (12%) longer mentions. These 268 discharge summaries were randomly assigned to a train (70%), development (15%) and test set (15%). We kept only the drug name annotations, and rejected the dose annotations and other drug-related information.

We also used a randomly sampled subset of the i2b2-2009 train set, to include the same number of drug name entities than the APcNER corpus, for quantitative comparison purposes. We call this smaller corpus "i2b2-small."

APcNER corpus

Document selection

We randomly sampled (stratified on document types) 147 documents from the dataset used for de-identification at AP-HP, excluding prescriptions and admission reports. The AP-HP de-identification dataset is a set of 3,223 French-language medical documents sampled (with upsampling of rare documents types) from over 50 million documents from the AP-HP data warehouse, which included EHR data from 39 hospitals. The APcNER included 4 main types of documents: discharge summaries, letters from physicians, operating reports and additional examination reports. Detailed document types can be found in the supplementary Table S1.

Annotation process

We based our annotation guideline on the UMLS® semantic types [START_REF] Bodenreider | The Unified Medical Language System (UMLS): integrating biomedical terminology [Internet[END_REF], Table 1 details the 5 medical entities that we annotated (Drug names, Signs or symptoms, Diseases or disorders, Diagnostic procedures or lab tests and Therapeutic procedures). We used BRAT Rapid Annotation Tool (BRAT) [START_REF] Stenetorp | BRAT: A Webbased Tool for NLP-assisted Text Annotation[END_REF]. The general guidance for annotation was to annotate the most complete entities (e.g., "Non-ST segment elevation myocardial infarction" and not only "myocardial infarction") with no possible overlap between entities (see Appendix Section 2 for the detailed annotation guideline). Documents were annotated by groups of 10 and the annotation time monitored. Documents were pre-annotated using a terminology-based annotator (see below). IL, a medical resident, annotated all the dataset. Then, in order to estimate the quality of the guidelines, we randomly selected 10 documents that NG, a medical doctor, annotated blindly. We then assessed the agreement between the two annotators as F-measures for each class, considering IL as gold standard, and the overall Cohen's Kappa, computed with exclusion of the outside class. Conflicting annotations have been discussed between IL and NG and are referenced in the Annotation Guideline (see Appendix Section 2). IL went through all the documents a second time in order to disambiguate the conflicting annotations.

For homogenization purposes, we fitted a simple conditional random field (CRF) model to the dataset with the default NER features, using Wapiti, an efficient open-source library for non-neural sequence labelling algorithms [START_REF] Lavergne | Practical Very Large Scale CRFs[END_REF]. We used this model to detect annotation inconsistencies or case errors, and manually corrected them. Finally, we randomly divided all the documents into 6 folds by stratifying on the types of documents and the length of the documents. The corpus can be made available on condition that a research project is accepted by the scientific and ethics committee of the AP-HP health data warehouse (https://recherche.aphp.fr/eds/recherche/).

Results of the annotation process

The first round took on average 87 min per 10 documents and a total of 21 hours. The second round took on average 28 min per 10 documents and a total of 7 hours. Both rounds took on average 115 minutes per 10 documents and a total of 28 hours. Overall, the 147 documents of the corpus contain 80,421 tokens, 4,837 entities and 3,093 sentences, for a vocabulary of 12,523 tokens. The median sentence length was 14. The inter-annotator agreement after the first round and before the CRF harmonization is reported in Table 2.

Cohen's Kappa was substantial for non-exact match ( = 0.61) and moderate considering exact match ( = 0.42).

Clinical-NER systems

For all experiments, we used the inside, outside, beginning (IOB) tagging scheme [START_REF] Ramshaw | Text Chunking Using Transformation-Based Learning[END_REF]. For an entity of type DRUG, the first token of such entity is coded B-DRUG, if the entity is constituted of multiples tokens, the following tokens are coded I-DRUG, and all tokens outside entities are coded O. For instance, "placed on heparin sodium" is encoded "O O B-DRUG I-DRUG."

Terminology based system

In English, we extracted drug names using regular expressions from UMLS® (including SNOMED 3.5 CT®) to create a large dictionary of drug names. In French, we used 10 terminologies, 8 of which were previously referenced in [START_REF] Névéol | Language Resources for French in the Biomedical Domain[END_REF] (ATC, BPDM, CCAM, CIM-10, DRC, SNOMED, UMLS), and 2 terminologies held by AP-HP (GLIMS, QDOC). Table 1 details the extracted UMLS semantic types by entity type. Terms were extracted using minimal regular expression rules, and then tokenized using Stanford CoreNLP [START_REF] Manning | The Stanford CoreNLP natural language processing toolkit[END_REF]. We discarded common terms based on Wikipedia word count. The matching rules were based on the apache-UIMA framework, CoreNLP and dkPRO and allowed multiple words matching, stop words, accent normalization and case insensitive matching. The source code is available with the GLP-3 license (https://github.com/EDS-APHP/uima-aphp/tree/master/uima-dict). Resolution of conflicting (overlapping) entities was done by randomly picking one of the conflicting entities.

Supervised system

Sentence segmentation and tokenization were performed using Stanford CoreNLP [START_REF] Manning | The Stanford CoreNLP natural language processing toolkit[END_REF].

Numbers were normalized to a unique token. We learned a biGRU-CRF (Bidirectional -Gated Recurrent Unit -Conditional Random Field) [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF], based on the NCRF++ implementation [START_REF] Yang | NCRF++: An Open-source Neural Sequence Labeling Toolkit[END_REF]. The model takes 2 types of inputs. First, English or French word embeddings trained with the Skip-Gram model [START_REF] Mikolov | Efficient Estimation of Word Representations in Vector Space[END_REF], from 2 million AP-HP documents for the French version (dimension 200), and 2 million MIMIC [START_REF] Johnson | MIMIC-III, a freely accessible critical care database[END_REF] clinical notes for the English version (dimension 100). Second, character embeddings processed by 1-dimensional convolution (kernel size 3) with max-pooling. The global token representation is the concatenation of the word and character embeddings (see Figure 1). The sequence of token representation is then processed forward and backward by the biGRU, which outputs an emission probability score for each entity class. Finally, the CRF decodes the sequence of labels by associating the emission probability score with a transition probability score (see Figure 2). We used a dropout rate of 0.5, an L2-norm on the model weights and early stopping to prevent overfitting. We used Bayesian optimization [START_REF] Uzuner | Extracting medication information from clinical text[END_REF] to perform hyperparameter tuning of the architecture (number of layers, number of neurons, character embedding dimension), learning rate and L2-norm. Note that with 1 entity type, NER is a 3 class classification problem, and with k entity types it is a k x 2 + 1 classification problem, e.g., with the 5 entities types of APcNER there are 11 classes to predict.

Hybrid system

We proposed a hybrid system in which a supervised model is associated with a terminologybased model. For each token, we added a feature representing the class predicted by the terminology-based system described above, which is then encoded as a categorical embedding of dimension 5. This embedding is then concatenated to the word embedding from the supervised system. This terminology based feature can take two values per entity types (e.g., B-Drug Name; I-Drug name), as well as one value for the "Outside" class. We also added a context feature based on the terminology of section titles that we have developed internally. The French section headings terminology was created based on documents of the same distribution as the APcNER corpus, the English section heading terminology was created for the 2018 AP-HP Datathon based on MIMIC notes. For each token, the context feature was the class of the last section heading, following the order of the document. Then, the context feature is encoded as a categorical embedding of dimension 5.

This embedding is concatenated to the word embedding from the supervised system and the other feature embedding. These feature embeddings are initialized randomly and learned during the optimization procedure.

Evaluation methodology and metrics

First, we evaluated the systems on the i2b2-2009 corpus, its reduced version i2b2-small, and on the APcNER corpus with labels limited to drug names. Then we evaluated the systems on the entire APcNER corpus, as one multi-class task. We compared the models based on F-measure, precision and recall using the CONLL definition: "precision is the percentage of named entities found by the learning system that are correct. Recall is the percentage of named entities present in the corpus that are found by the system. A named entity is correct only if it is an exact match of the corresponding entity in the data file." We also compared the model based on partial match, which allowed the boundaries of the entity to mismatch.

Neural network models' training is highly non-deterministic and is subject to the random seed choice. Because of this variability during the training phase, we performed five experiments for each model presented in this work, and reported the mean, and 95% confidence interval assuming a t-distribution of each metric. Note that such confidence interval only accounts for the variability originated from the optimization procedure, and not the variability originated when sampling the documents from the data warehouse.

First, using default parameters of NCRF++, we tested the main features of the architecture, ablation of the CRF, GRU versus LSTM, ablation of character embeddings, and different set of fast-text and skip-grams embeddings. This first step led to the selection of our baseline architecture, the biGRU-CRF with skip-gram and character embeddings. Second, we performed fined tuning of this architecture. For the i2b2-2009 corpus, we selected the optimal set of hyperparameters for the supervised model based on a development set, including the optimal epoch stop, and evaluated on the test set the models trained on the train+dev set. For APcNER, we selected the optimal set of hyperparameters for the supervised model by 6 fold cross-validation, we defined the optimal epoch stop as the mean of the optimal epoch stop for each fold. We evaluate the model on each fold, after training it on the remaining 5 folds, using the same set of optimal hyperparameters. We then report the evaluation metrics computed over the 6 folds.

Results

Table 3 (Diagnostic procedure or lab test) and 61.3% versus 16.6% (Therapeutic procedure). The difference between exact-match and partial match metrics is also greater than for Drug names. As illustrated in Figure 3, the hybrid system outperforms the other systems for all entity types except Therapeutic procedure. Table S2 summarizes the hyperparameters of the models.

Discussion

In this study, we built APcNER, a corpus for clinical-NER of 5 types of entities (Drug names, Signs or symptoms, Diseases or disorders, Diagnostic procedures or lab tests and Therapeutic procedures). We then systematically evaluated a supervised model (biGRU-CRF) against a terminology-based system. Finally, we proposed to extend the supervised system by encoding the predictions of the terminology-based system as categorical embeddings. This hybrid model learns from both inputs: word embeddings carrying semantic and syntactic information from words, and terminology-based embeddings carrying information from external resources. Thus, the powerful modelling capability of the biGRU allows the system to learn from the noisy terminology-based predictions. On the APcNER and on the i2b2-2009 corpora, the biGRU-CRF outperformed the terminology-based system.

The hybrid system was more efficient in low regime of entities, for APcNER (except for the Therapeutic procedure class) and i2b2-small.

Both the biGRU-CRF and its extended version outperform previous results from the i2b2

2009 Medication Challenge (90% F-measure for the best team) [START_REF] Uzuner | Extracting medication information from clinical text[END_REF]. These results (mean 92.2 [91.4-92.9]) are very close to FABLE [START_REF] Tao | FABLE: A Semi-Supervised Prescription Information Extraction System[END_REF] which used bootstrapping, a semi-supervised approach, leading to 93% F-measure. As the number of examples increases, the information brought by the terminology should become redundant with the one brought by the annotations, which could explain the relatively larger performance gain of the hybrid system in low regime of trained examples (see Table 3). Note that, as mentioned above, our test set is a sub-sample of the i2b2 test set used during the challenge.

As illustrated in Figure 3, the performance gain of the hybrid system from the purely supervised system appears to be proportional to the performance of the terminology-based system, and inversely proportional to the performance of the supervised system. However, the actual relation is more complex, and the performance of the hybrid system for Signs and Symptoms does not fit with this hypothesis. The number of co-occurrence terms between terminologies could shed light on this outlier, as the Signs and Symptoms category has much more co-occurrence term (15 ‰) than the other entities. Hence, the hybrid system gain for this entity could be boosted by the features from other categories.

The difference in performance for the biGRU-CRF between the i2b2-2009 and the APcNER corpora is partially explained by their difference in number of annotated entities. Indeed, our results (see Table 3) on the reduced version of i2b2 show close performance with the hybrid system when the training set is reduced to the same number of entities than APcNER.

Differences remain with the system using no external resources, but this may be due to the fact that the scope covered by the APcNER corpus is much broader in terms of document types and medical specialties. Another noteworthy difference is that a drug name followed by its commercial name between brackets is annotated as a single entity in i2b2-2009, but as several separate entities in APcNER, which explains the difference in the distribution of long n-grams (n ≥ 3) between the two corpus (see Table 2).

Another notable result is that the performance of the biGRU-CRF is much lower for other types of entities than Drug names, and the difference between exact match and partial match is larger (see Table S4). Along with the results of the APcNER annotation process (Table 2), it suggests that it is partly due to the longer size of the entities. Indeed, compared to Drug names, other categories have between 3 and 6 times more entities composed of at least 2 tokens. The inter-annotator agreement is also lower for these types of entities, and the results of the biGRU-CRF are consistent with those of the inter-annotator agreement.

Moreover, it is also consistent with feedback from the annotation process that boundaries are more difficult to define for longer entities. In addition, the conflict between overlapping entities rarely concerns Drug names, whereas they are more likely to occur between Diagnostic and Therapeutic procedure (e.g., angiography), or between Disorder and Sign (e.g., hemiparesis). Following this analysis, we argue that for the APcNER corpus, the metric of reference should be the non-exact F-measure for entities other than Drug names.

In comparison with MERLOT [START_REF] Campillos | A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)[END_REF], which include 44,740 entities of 12 types, for 500 documents from Hepato-gastro-enterology and Nutrition ward, APcNER is both smaller (147 documents) and covering a broader scope (no restriction of medical specialty). The interannotator agreement for class common to both corpora are comparable, with Signs and Symptoms exact F-measure 59% versus 55%, Drug names 90% versus 85%, and Diseases and disorders 77% versus 65% for MERLOT and APcNER, respectively.

To our knowledge, our study is the first to provide an estimation of the annotation efficiency for clinical-NER for a distribution of medical documents that is representative of that of an EHR (with the exception of imagery reports and drug prescriptions). The annotation efficiency estimated is achieving on average 84% of non-exact match F-measure for the cost of 28 hours of annotations. Using active learning is likely to diminish this cost by 40 to 80% [START_REF] Shen | Multi-criteria-based Active Learning for Named Entity Recognition[END_REF][START_REF] Chen | A study of active learning methods for named entity recognition in clinical text[END_REF], hence reaching performance greater than 95% on this task seems possible. In addition, we found consistent results in English and French, which provide an insight into the complementarity of a terminology with a supervised model.

The main limitation of our study is the small size of our corpus compared to the broad scope it covers. However, using cross-validation allowed us to maintain comparable regime of entities with the test set of other corpora. If cross-validation is known to present a risk of overfitting [START_REF] Ng | Preventing" overfitting" of cross-validation data[END_REF], we did not tune the hyperparameters for the hybrid systems, hence the performance gain relative to the biGRU-CRF is a lower bound. Finally, in regards of the average low performance of the supervised model on APcNER (average F-measure of 67.1%), one could think the corpus unfit to allow for supervised learning. However, it still constitutes an important tool to evaluate semi-supervised or unsupervised systems. Indeed, large pre-trained language models such as XLNET [START_REF] Yang | XLNet: Generalized Autoregressive Pretraining for Language Understanding[END_REF] could help overcome this low annotation regime by providing better contextualized word representation, which is therefore a direction that we will explore in future research. In addition, combined with a more focused dataset (such as MERLOT [START_REF] Campillos | A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)[END_REF]), it could allow interesting transfer learning approaches that are to be tested.

Conclusion

APcNER is a French corpus for clinical named-entity recognition of five types of entities which covers a large variety of document types. The extension of the supervised model with terminology has allowed an easy increase in performance, especially for rare entities.

FiguresFigure 1 .

 1 FiguresFigure 1. Word representationThe word representation is the concatenation of a word embedding, a character embedding and a terminology-based feature embedding.

Figure 2 .Figure 3 .

 23 Figure 2. biGRU-CRF architecture

  summarizes the results of Drug name recognition in the i2b2-2009 corpus, i2b2small (reduced version of the former dataset) and the APcNER corpus. For both i2b2-2009

	and APcNER, the biGRU-CRF outperforms the terminology-based system, with an exact-
	match F-measure of 91.1% [95% CI, 90.3-91.9] versus 73% and 81.9% [81.2-82.6] versus
	75% respectively. For both i2b2-small and APcNER, the hybrid system outperforms the
	biGRU-CRF, with an exact-match F-measure of 87.8% [86.4-89.2] versus 85.6% [84.8-86.3]
	and 88.4% [86.1-86.7] versus 81.9% [81.2-82.6] respectively. The performance on i2b2-
	small is very close to the performance on APcNER for the hybrid system (with an exact-
	match F-measure of 87.8% and 86.4%).
	Table S4 summarises the results of clinical-NER on all the entity types of the APcNER

corpus. For all three systems, the exact-match performance for Sign or symptom, Disease or disorder, Diagnostic procedure or lab test, and Therapeutic procedure are much lower compared to Drug name. The difference between the biGRU-CRF and the terminologies are also more important than for Drug names, with exact-match F-measures of 55.2% versus 15.7% (Sign or symptom), 59.5% versus 30.9% (Disease or disorder), 75.9% versus 30.4%

Table 1 . UMLS semantic types extracted for each entity Entity types

 1

	O	O	B-DRUG

Table 2 . APcNER, a french corpus for five clinical entities.

 2

			F-measure [95% CI]
	Corpus	System	Exact-match	Partial-match
	i2b2-2009 Terminologies 73	84.6
		biGRU-CRF	91.1 [90.3-91.9]	93.5 [92.7-94.3]
		Hybrid system 92.2 [91.4-92.9]	94.7 [94.1-95.2]
	i2b2-small biGRU-CRF	85.6 [84.8-86.3]	90.4 [89.8-91.1]
		Hybrid system 87.8 [86.4-89.2]	90.6 [86.7-94.6]
	APcNER	Terminologies 75	77.7
		biGRU-CRF	81.9 [81.2-82.6]	86.4 [85.3-87.6]
		Hybrid system 86.4 [86.1-86.7]	90.4 [89.8-90.9]

Table 3 . Drug name recognition. ...USING KNOWLEDGE FROM TERMINOLOGY LEARN CONTEXT CUES... ... TO LABEL DRUG NAME...

 3 Patient was placed on sodium heparin before surgery.

Note that the original test set used during the i2b2 challenge is made of the

collectively annotated documents.

Acknowledgements

We thank Nicolas Griffon for taking part in the annotation process. We thank Christel Daniel for resources and review of the manuscript. We thank Raphaël Veil for proofreading the manuscript.

Availability of data and material:

The code made for simulations is available at https://gitlab.com/lerner.ivan/apcner. This project was accepted by the scientific and ethics committee of the AP-HP health data warehouse as CSE-18-0035. The corpus can be made available on condition that a research project is accepted by the scientific and ethics committee of the AP-HP health data warehouse (https://recherche.aphp.fr/eds/recherche/).

Tables Table 1

. UMLS semantic types extracted for each entity Table 2. APcNER, a French corpus for 5 clinical entities

The F-measure is the harmonic mean between precision and recall computed as in Conll 2003. Interannotator agreement is evaluated on a random subset of APcNER of 10 documents.

Table 3. Drug name recognition

Comparison between a terminology-based system, a supervised model (biGRU-CRF) and a hybrid system on: the i2b2-2009 corpus, i2b2-small (a reduced version of the former corpus), and APcNER.

We report the mean over 5 runs for exact match and partial match, along with its 95% confidence interval assuming a t-distribution.


Randa Bittar

Alain Carrié

Georges Nouadje

Corinne Cherfils

Valérie Fesel-Fouquier

Laurence Barbot-Trystram

Philippe Giral

Dominique Bonnefont-Rousselot

Pr Dominique Bonnefont-Rousselot

Evaluation of a semi-automatic isoelectric focusing method for apolipoprotein E phenotyping

Keywords: apolipoprotein E, isoforms, phenotype, isoelectric focusing apo : apolipoprotein, HDL: high density lipoprotein, IDL : intermediate density lipoprotein, IEF : isoelectric focusing, LDL: low density lipoprotein, VLDL : very low density lipoprotein

A qualitative, semi-automatized method for apolipoproteinE (apoE) phenotyping by isoelectric focusing method has been evaluated on 40 serum samples from patients previously genotyped for apoE, especially as regards concordance with genotyping, but also repeatability and reproducibility of the method, and sample storage. Total concordance with genotyping and good precision criteria, together with its practicability and requirement of a little sample volume, lead to conclude to the usefulness of this method to help clinicians in the diagnosis of dyslipidemic and neurodegenerative diseases.

Introduction

Apolipoprotein E (apoE) is a 34-kDa glycoprotein of 299 amino acids, encoded by the human APOE gene located on chromosome 19. Three alleles of this gene occur at different frequencies in humans : ε2 (5-10%), ε3 (65-70%) and ε4 (15-20%), leading to three homozygous (apoE2/E2, apoE3/E3, apoE4/E4) and three heterozygous (apoE3/E2, apoE4/E2, apoE4/E3) phenotypes [START_REF] Mahley | Apolipoprotein E: from cardiovascular disease to neurodegenerative disorders[END_REF]. The three isoforms produced, i.e., apoE2 (cys112, cys158), apoE3 (cys112, arg158) and apoE4 (arg112, arg158), differ by the structure (conformation, isoelectric point, molecular weight) and by their functional properties [START_REF] Huang | Apolipoprotein E: structure and function in lipid metabolism, neurobiology, and Alzheimer's diseases[END_REF]. Plasma apoE mainly arises from liver hepatocytes (75%), but the brain (astrocytes, oligodendrocytes, microglia, astrocytes) also synthesizes apoE found in the cerebrospinal fluid. ApoE is a major component of lipoproteins which participates in the transport and clearance of lipids. ApoE4 status is a risk factor for Alzheimer's and other neurodegenerative diseases, whereas apoE2 and also apoE4 increase the risk for cardiovascular disease [START_REF] Mahley | Apolipoprotein E: from cardiovascular disease to neurodegenerative disorders[END_REF]. From a practical standpoint, determination of apoE status using isoelectric focusing (IEF) can be easier than genotyping (the latter requiring a genetic consent) and thus constitutes a valuable alternative to genotyping. Initial IEF methods required isolation of lipoproteins (i.e, very low density lipoproteins, VLDLs, and intermediate density lipoproteins, IDLs) by ultracentrifugation and their delipidation before performing isoelectric focusing in a pH 4-6 ampholine gradient [START_REF] Warnick | Gel isoelectric focusing method for specific diagnosis of familial hyperlipoproteinemia type 3[END_REF]. A twodimensional electrophoresis [START_REF] Sprecher | Two-dimensional electrophoresis of human plasma apolipoproteins[END_REF] could further allow confirming E2/E2 phenotype or detection of a mutated form of apoE. Despite that the apoE phenotype revealed by isoelectric focusing was established with more than 20 years ago, a kit commercially available for clinical laboratories does not exist. More recently, a semi-automatic method has been developed that can be performed directly on serum and that was stated as concordant with mass spectrometry [START_REF] Hirtz | Development of new quantitative mass spectrometry and semiautomatic isofocusing methods for the determination of apolipoprotein E typing[END_REF]. The present study aimed at evaluating this simple qualitative, semi-automatic apoE IEF method on selected samples from patients previously genotyped for apoE, especially regarding concordance with apoE genotyping, but also repeatability, reproducibility and sample storage criteria.

Material and Methods

In the course of familial hypercholesterolemia screening or mixed dyslipidemia diagnosis, blood samples were collected from 40 patients in our outpatient clinic (Cardiovascular Prevention Unit, Institute of Cardiometabolism and Nutrition, La Pitié-Salpêtrière -Charles Foix University Hospital, AP-HP, Paris, France) of the Endocrinology-Metabolism Department. These patients had given their informed consent including genetic determination and APOE genotypes were already determined by Sanger sequencing of a PCR fragment encompassing the two polymorphic sites (rs429358 and rs7412). Venous blood was collected in gel-containing Vacutainer® tubes (Becton-Dickinson, Plymouth, UK), then allowed to clot at room temperature and centrifuged at 4,500 rpm at 10°C for 10 minutes, to obtain serum for routine lipid parameters analysis. Sera were immediately aliquoted and frozen at -80°C until IEF assays. The Hydragel 18 Apo E Isofocusing (Sebia) is a qualitative RUO (Research Use Only) kit for detection and identification of the different apoE phenotypes. The method is adapted on a well-established equipment from Sebia. The references indicated on the apoE kit are temporary references since they are currently RUO products. A ready-to-use agarose gel containing ampholytes (pH gradient : 5-8) is used to perform a semi-automatic electrophoresis on Hydrasys 2 Scan Sebia (Lisses, France), followed by a specific immunofixation with anti-apoE antiserum. Isoform bands were identified by comparison to bands of a specific control run in each gel and including E2, E3, and E4 phenotypes (Sebia, PN 4785). After reconstitution of this lyophilized control, aliquots were kept frozen at -80°C. The Hydrasys 2 Scan Sebia can perform all steps of process (migration, incubation, gel staining/destaining, drying and scanning). The "apoE isofocusing visualization" kit (Sebia, PN 4749) included an antiserum diluent, stock solutions of anti-apoE antiserum and peroxidase-labeled antibody, and reagent for revelation (TTF1 and TTF2 developing solutions). Briefly, 20 µL serum were treated with 5 µL neuraminidase (neuraminidase type V from Clostridium perfringens, provided in the kit: Sigma, PN N2876) for 1h at 45°C. Sample was then delipidated with 25 µL of the delipidation solution for 1h at 45°C. After centrifugation for 5 min. at 5,000 rpm, the supernatant was collected and used for IEF. Eighteen 10 µL-samples could be analyzed on each gel. Migration was carried out under 500 V until 300 Vh has been accumulated and under 1000 V until 100 Vh has been accumulated (final 400 Vh), at 20°C (temperature controlled by Peltier effect) for about 45 min. A two step-immunofixation was then performed: one with the anti-human apoE polyclonal antiserum for 10 min. at 20°C (temperature controlled by Peltier effect) and, after a cycle wash, another one for 10 min. with a peroxidase-labeled secondary antibody. The gel was then incubated with the Sebia TTF1/TTF2 substrate for 10 min. in order to visualize apoE isoforms. The total time to obtain the apoE status on a gel was 2h15. The 40 sera were blindly analyzed for determination of apoE status. The repeatability of the procedure was evaluated by performing IEF of each patient sample 6-fold in the same gel. For reproducibility, each patient sample was tested 4-fold on different gels and at different days, which implied that stability of sera was tested after 3 cycles of freezing and thawing. The Sebia "control serum" (E2/E3/E4) was run in each gel, together with two sera from already typed patients.

Results

The sera from the genotyped patients included in the study allowed us to test the following isoforms: E3/E3 (18 sera), E3/E4 (16 sera), E2/E2 (4 sera) and E4/E4 (2 sera) (table 1). All phenotypes tested were 100% concordant to the genetic isoforms, even if all the possible phenotypes are not found in our population. An IEF gel is shown as an example (Figure 1).

Discussion

This study allowed us to show the advantages of this semi-automated procedure to perform apoE phenotyping, which presents an interest under some conditions where no genetic material is available. Its good precision criteria and the possibility to use frozen samples (up to 3 freezingthawing cycles) must be put forward for the lab practice.

Our patient population differs from that of the Hirtz's study [START_REF] Hirtz | Development of new quantitative mass spectrometry and semiautomatic isofocusing methods for the determination of apolipoprotein E typing[END_REF]. Indeed, the latter population was constituted of patients with cognitive or behavioral disorders and presented with a high frequency (9.3%) of E4/E4 phenotypes. By contrast, in agreement with our recruitment characteristics, the patients of our study were at cardiovascular risk and 10% presented with E2/E2 phenotype, one situation that is known to be a risk factor to develop type III dyslipidemia (dysbetalipoproteinemia), with high levels of circulating intermediate density lipoproteins (IDLs) due to a decreased clearance of these lipoproteins by the liver [START_REF] Mahley | Pathogenesis of type III hyperlipoproteinemia (dysbetalipoproteinemia): questions, quandaries, and paradoxes[END_REF]. ApoE4 also increases the risk for cardiovascular disease due to its structure which displays increased preference for large, triglyceride-rich VLDLs. In a whole, these features could explain the high frequency of E2/E2 and E3/E4 phenotypes (10 and 40%, respectively) in our population when compared with normolipidemic subjects or in general population ( [START_REF] Poirier | Apolipoprotein E polymorphism and Alzheimer's disease[END_REF][START_REF] Neuman | L'apolipoprotéine E dans la maladie d'Alzheimer : point de vue clinique[END_REF], supplemental Table 1).

When compared with the initial apoE IEF procedures, the direct use of serum instead of isolated lipoproteins represents a time spare and a technical facilitation. The readability of the gel is also increased owing to the action of neuraminidase that eliminates bands corresponding to the sialylated forms of apoE, and the position of the band of each isoform is especially easy to determine by comparison with the "normal control serum" run in each gel. Another advantage is the direct immunofixation step without requiring transfer to a membrane. Finally, good repeatability and reproducibility, and the possibility to perform 3 freezing-thawing cycles on serum samples support the reliability and practicability of the test. Thanks to our experience of the kit, we have shown a limited conservation of the neuraminidase provided in the kit (not more than 3 weeks) as well as the "normal serum" Sebia whose bands become less clear over time. For the latter reason, it is very important to include 2 pre-genotyped sera in each run to ensure proper identification of the position of the E2, E3 and E4 bands.

Other phenotyping methods are available, such as mass spectrometry [START_REF] Hirtz | Development of new quantitative mass spectrometry and semiautomatic isofocusing methods for the determination of apolipoprotein E typing[END_REF] and ELISA. Nevertheless, in opposition with mass spectrometry that is able to detect all isoforms with a high specificity, ELISA techniques are specific for apoE4, thus not allowing a complete apoE phenotyping [START_REF] Uchida | Sandwich ELISA for the measurement of Apo-E4 levels in serum and the estimation of the allelic status of Apo-E4 isoforms[END_REF]. A recent method based on apoE-polystyrene interaction has been described, but it is again specific for apoE4 detection and does not distinguish between homozygote E4/E4 or heterozygous phenotypes containing E4 isoform [START_REF] Calero | A fast and cost-effective method for apoE isotyping as an alternative to apoE genotyping for patient screening and stratification[END_REF].

Due to the specificity of the present technique (in relation with immunofixation), it is of note that other apolipoproteins present in VLDLs and IDLs cannot be seen, especially apoC (C-I, C-II, C-III), whereas they were of course present in the initial procedure using isolated lipoproteins and without immunofixation.

A limitation of our study is the low number of patients and phenotypes tested (especially, no mutated isoform was included), but the patients were selected for having had a previous genotyping and for a perfect quality and storage of their frozen sera. The impact of the sample matrix has also been tested on 4 patients, for which frozen EDTA or heparin plasma was available; only EDTA plasma gave correct apoE phenotyping, while heparin plasma has to be avoided under our experimental conditions, because of unclear interpretation (data not shown). Nevertheless, for the latter samples, storage was not well controlled before samples were frozen, so that we cannot rule out preanalytical deficiencies. Of note, Hirtz et al. [START_REF] Hirtz | Development of new quantitative mass spectrometry and semiautomatic isofocusing methods for the determination of apolipoprotein E typing[END_REF] did not report such interference on heparin samples.

Conclusion

This qualitative, semi-automatized method could yield a novel and simple tool to phenotype apoE isoforms. Validated on a cohort of patient samples with known genotype, it could be used either for research screening and stratification of a patient cohort, or for clinical analysis when genetic material cannot be obtained. Another advantage for this test is the possibility to use a little volume of routine sera, and practice serial analysis in a short time with simplified and standardized steps. This could be a useful and supplementary tool helping the clinician in the diagnosis of dyslipidemic and neurodegenerative diseases.

Figure 1 :

 1 Figure 1 : Hydragel 18 Apo E Isofocusing (Sebia) showing different apoE isoforms from 16 patients tested. The "control serum" provided in the Sebia kit is included in each series, together with two already typed serum samples as internal controls (as an example, E3/E2 and E4/E4).Precision of the technique was verified with the repeatability (supplemental Figure1) and the reproducibility tests for the different isoforms (data not shown), since tests gave similar results. In addition, the reproducibility tests led us to perform three freezing-thawing cycles on serum samples and a 100% concordant interpretation with genotyping was maintained. Supplemental Figures2 and 3show no interference of hemolysis, glucose or hypertriglyceridemia at the respective concentrations of 45.3 µmol/L hemoglobin, 25.1 mmol/L glucose and 9.5 and 4.8 mmol/L triglycerides for the determination of the phenotype profile. Supplemental Figures 2 and 3 also show various apoE phenotypes.

Table 1 :

 1 Lipid parameters and apoE status of the patients.

	Patients n = 40
	Gender (M/F)	20/20
	Age (years)	52 ± 13
	Total cholesterol (mmol/L)	5.9 ± 1.3
	Triglycerides (mmol/L)	2.0 ± 1.3
	HDL-cholesterol (mmol/L)	1.37 ± 0.47
	LDL-cholesterol (mmol/L)	3.65 ± 1.24
	ApoA1 (g/L)	1.55 ± 0.26
	ApoB (g/L)	1.10 ± 0.33
	Lp(a) (g/L)	0.36 ± 0.38
	Phenotype	Frequency (%) Genotype
	E3/E3	45	ε3/ε3
	E3/E4	40	ε3/ε4
	E2/E2	10	ε2/ε2
	E4/E4	5	ε4/ε4

Acknowledgments

The authors would like to express special thanks to Sebia for providing the kits to perform the evaluation and for technical improvement of the method.

Supplemental Figure 1 : Example of Hydragel 18 Apo E Isofocusing (Sebia) showing repeatability of apoE phenotyping performed on the "control serum" provided in the Sebia kit (E2/E3/E4) and on two patient's sera (E4/E4 and E3/E3) (n=6 for each sample). Supplemental Figure 2 : Example of Hydragel 18 Apo E Isofocusing (Sebia) with various apoE phenotypes and study of the interference of hypertriglyceridemia for the determination of the phenotype profile. A patient EDTA plasma sample (P) with E3/E3 phenotype has been loaded with glucose (G), or with hemoglobin (H) to assess the influence of hyperglycemia and hemolysis, respectively. To test glucose interference, a stock fresh solution of 5.4 mol/L glucose has been added to the plasma (initial glucose concentration : 4.4 mmol/L) to obtain a final glucose concentration of 25.1 mmol/L. To test hemolysis interference, the plasma sample has been loaded with hemoglobin (H) by adding an hemolysate obtained by inducing osmotic lysis of the patient's red blood cells by distilled water (v/v). The initial and final plasma hemoglobin concentrations were 8 and 45.3 µmol/L, respectively. Supplemental Figure 3 : Example of Hydragel 18 Apo E Isofocusing (Sebia) with various apoE phenotypes and study of the interference of hypertriglyceridemia for the determination of the phenotype profile. A patient serum sample (L) with E3/E4 phenotype and hypertriglyceridemia was diluted 1/1 (v/v) with 0.15 M NaCl (L/2). Initial and final triglyceride concentrations were 9.5 and 4.8 mmol/L, respectively.


Nicolas Girerd
email: n.girerd@chru-nancy.fr

Emmanuel Bresso

Marie-Dominique Devignes

Patrick Rossignol

Pr Nicolas Girerd

Insulin-like Growth Factor Binding Protein 2: A prognostic biomarker for heart failure hardly redundant with natriuretic peptides

Insulin-like Growth Factor Binding Protein 2 (IGFBP2) is a member of the IGFBP family which is present in the heart; cardiac IGFBP2 mRNA levels have furthermore been shown to rise in animal models of ischemia-induced heart failure (HF) [START_REF] Berry | Proteomics analysis reveals IGFBP2 as a candidate diagnostic biomarker for heart failure[END_REF] . The group of Dr. P. Rouet previously showed that IGFBP2 is a powerful diagnostic biomarker for HF whose accuracy for identifying acute HF is a valuable adjunct to brain natriuretic peptide (BNP) [START_REF] Berry | Proteomics analysis reveals IGFBP2 as a candidate diagnostic biomarker for heart failure[END_REF] : The area under the curve of the receiver operating characteristic (ROC) curve for HF was 0.908 (0.830-0.958) for IGFBP2 versus 0.857 (0.770-0.921) for BNP. In addition, IGFBP2 improved the model's accuracy in predicting HF as well as the net reclassification index (NRI) (0.574 (0.340-0.750) p<0.001) on top of BNP.

The group of Dr. Rouet now reports the results of an elegant study to assess the prognostic value of IGFBP2 in 870 patients with known HF from 3 independent international cohorts [START_REF] Barutaut | Insulin-like Growth Factor Binding Protein 2 predicts mortality risk in heart failure[END_REF] . In this study, their results

show that the highest quartile of IGFBP2 was associated with mortality in the Maastricht cohort and in the combined Atlanta and Toulouse cohorts with Hazard Ratios > 1.5. More importantly, adding IGFBP2 to survival models significantly increased the NRI for death in the 3 pooled cohorts on top of clinical variables (0.404, p=0.01 at 1 year, 0.315, p=0.02 at 3 years) as well as on top of clinical variables and BNP (0.404, p=0.0005 at 1 year, 0.285, p=0.01 at 3 years).

These results emphasize the vast potential of IGFBP2 in the field of HF both for diagnostic and prognostic purposes. However, our understanding of the exact role of IGFBP2 in HF is still lacking. The fact that IGFBP2 surprisingly and notably adds to the diagnostic and prognostic value of BNP suggests that IGFBP2 features a different pathophysiological background than natriuretic peptides. Yet, achieving a comprehensive © 2019 published by Elsevier. This manuscript is made available under the CC BY NC user license https://creativecommons.org/licenses/by-nc/4.0/ Version of Record: https://www.sciencedirect.com/science/article/pii/S0168822719310241 view on the current evidence regarding IGFBP2 and its link with natriuretic peptides and HF remains challenging: complex network analysis is an increasingly used [START_REF] Ferreira | Proteomic Bioprofiles and Mechanistic Pathways of Progression to Heart Failure[END_REF] analysis strategy allowing such integrated view of available evidence.

In order to gain a better understanding of the interplay between IGFBP2, BNP and HF, we used the graph knowledge box (GKBox) developed within the FIGHT-HF program (https://www.inicrct.org/en/ini-crctpartner-hospitalo-universitaire-rhu-fighting-heart-failure-fight-hf-research-program-9). The GKBox integrates data from over 20 public biological databases such as UniProt, DisGeNET or String. In this instance, the GKBox can be used to better understand the mechanisms underpinning the association between IGFBP2, BNP and HF by better positioning the biomarkers in their biological context and identifying underlying pathways or mechanism(s) of action.

Specifically, we built a network diagram (Figure ,panel A) representing proteins (in grey) associated with HF and directly linked to either IGFBP2 or BNP, and Gene Ontology biological processes connecting at least 2 proteins (in blue). From this diagram, it appears that there is no overlapping biological process emerging from both BNP and IGFBP2. In addition, the proteins emerging from these Gene Ontology terms are distinct between the BNP and the IGFBP2 components of the diagrams (respectively ANP and ANP receptors for BNP and Hypoxia-inducible factor 1-alpha [HIF1A] and Insulin-like growth factor 1 [IGF1] for IGFBP2). There are only 5 Gene Ontology terms linking the ANP/ANP receptor to IGFBP2/HIF1A/IGF1, most of which are hardly specific (response to hypoxia and to insulin, cellular protein metabolic process, signal transduction and female pregnancy). BNP is related to the Gene Ontology term associated with congestion (body fluid secretion, regulation of vascular permeability, positive regulation of renal sodium excretion) whereas IGFBP2 is currently predominantly related to the Gene Ontology evocative of glucose metabolism (regulation of glycolytic process) and cellular death (regulation of apoptotic process and activated T cell proliferation). Of note, such associations with hypoxia and apoptotic process echo previous findings of IGFBP2 being overexpressed in an experimental model of ischemic HF.

In a second more detailed network diagram (Figure, panel B), we plotted the proteins linked to HF and indirectly linked to IGFBP2 or BNP through a Gene Ontology biological process term (in blue). This large diagram allows gaining a more comprehensive view of current biological evidence linking IGFBP2/BNP with HF. In this panel, a number of proteins are related to both BNP and IGFBP2 (in the center of the diagram), some of which are associated with the renin angiotensin aldosterone system, while a large number of proteins are related to only IGFBP2 (on the right of the diagram). Additional Gene Ontology terms not included in the panel A diagram are considered in this diagram, showing that, in the context of HF, IGFBP2 is related to the response to mechanical stimulus, aging, as well as to the response to various hormonal stimuli (steroid, estrogens, estradiol, glucocorticoid, insulin).

From these complex network representations, it can be seen that there is little interconnection between BNP and IGFBP2, probably explaining why IGFBP2 conveys additional value on top of BNP. In addition, there is evidence of IGFBP2 being linked to mechanical stimulus, which may be underpinning its strong diagnostic value for acute HF. Lastly, the association of IGFBP2 with hormonal response and apoptotic mechanisms is possibly related to its good prognostic value in HF patients. It should nevertheless be emphasized that the lack of

Acknowledgements: NG, EB, MDD and PR are supported by the RHU Fight-HF, a public grant overseen by the French National Research Agency (ANR) as part of the second "Investissements d'Avenir" program (reference: ANR-15-RHUS-0004), by the French PIA project "Lorraine Université d'Excellence" (reference: ANR-15-IDEX-04-LUE), the ANR FOCUS-MR (reference: ANR-15-CE14-0032-01), ERA-CVD EXPERT (reference: ANR-16-ECVD-0002-02), Contrat de Plan Etat Lorraine IT2MP and FEDER Lorraine.

currently reported in the literature (but could be in the future following additional research).

All in all, IGFBP2 is a promising biomarker for HF which needs further investigation. The prognostic value of changes in IGFBP2 in HF patients could further strengthen the evidence supporting its clinical use, along with natriuretic peptides and other biological and imaging markers of congestion [START_REF] Kobayashi | Chest X-ray quantification of admission lung congestion as a prognostic factor in patients admitted for worsening heart failure from the ICALOR cohort study[END_REF][START_REF] Kobayashi | Mid-term prognostic impact of residual pulmonary congestion assessed by radiographic scoring in patients admitted for worsening heart failure[END_REF][START_REF] Girerd | Integrative Assessment of Congestion in Heart Failure Throughout the Patient Journey[END_REF][START_REF] Hubert | Diagnostic accuracy of lung ultrasound for identification of elevated left ventricular filling pressure[END_REF] . Obviously, as for any HF biomarker, designing trials tailoring HF medication to repeat IGFBP2 measurements would indubitably represent the ultimate proof of its usefulness in the management of HF.