Keywords: Natural language processing, Expert knowledge, Domain adaptation, Neural networks, Deep learning, Transformers

This thesis is the fruit borne by the tree grown from four years of labor, over the course of which I have invested more of myself than I knew I had to offer. Despite the caffeine addiction and sleep lost, despite the hardships and repetitive strain injuries sustained, despite the paperwork and relationships neglected, I look back on this section of my life with few regrets, for life can only thrive under stress. However, much like a tree only stands strong against the storm thanks to its roots, I have my own roots to thank for supporting me throughout this shared skybound journey.

I would first like to thank my parents, the brightest minds I've ever had the chance to probe, and whose legacy I can only hope to live up to one day. To my father, in whose footsteps I seem to have followed despite my best efforts, I am grateful for introducing me to computer science and machine learning, as well as for his support in these shared interests. To my mother, I am grateful for instilling in me the value of hard work, and for teaching me to write, particularly in our elegant mother tongue. To both, I am especially grateful for their continuous moral and logistical support, without which I could never have come this far.

I would also like to thank my friends, who have stood by me despite my occasional lack of care in maintaining our bonds. In particular, I would like to thank my dear friend and sister in arms, Wafa Aissa, without whom I would not have had the opportunity to pursue this doctorate.

I am grateful to my colleagues from the CEA for the positive and laid back work environment they fostered, the interest they took in my work, and the valuable insights they gave.

Lastly, I would like to thank my supervisors and doctoral committee. In particular, I would like to extend my utmost gratitude for the continuous guidance and encouragement of my main supervisor and mentor, Nasredine Semmar. To Julien Tourille, I am grateful for consistently taking time out of his busy schedule to lend me his technical expertise in all things biomedical and knowledge-graph-related. I take this opportunity to acknowledge Hassane Essafi who, unsolicited, took interest in my work to the point of insisting on taking part in every meeting, and helped come up with many new ideas and directions for research. I would like to thank Profs. Rapp and Tannier for agreeing to review this dissertation. Finally, I would like to extend my thanks to my director, Alexandre Allauzen, for his supervision, often in the form of characteristically specific, pertinent and insightful suggestions, as well as for his aid in having me selected as a part-time teaching assistant at Dauphine

Abstract: Current state-of-the-art Language Models (LMs) are able to converse, summarize, translate, solve novel problems, reason, and use abstract concepts at a near-human level. However, to achieve such abilities, and in particular to acquire "common sense" and domain-specific knowledge, they require vast amounts of text, which are not available in all languages or domains. Additionally, their computational requirements are out of reach for most organizations, limiting their potential for specificity and their applicability in the context of sensitive data.

Knowledge Graphs (KGs) are sources of structured knowledge which associate linguistic concepts through semantic relations. These graphs are sources of high quality knowledge which pre-exist in a variety of otherwise low-resource domains, and are denser in information than typical text. By allowing LMs to leverage these information structures, the burden of memorizing facts can be removed from LMs, potentially reducing the amount of text and computation required to train them and allowing us to update their knowledge with little to no additional training by updating the KGs, therefore broadening their scope of applicability and making them more democratizable.

Various approaches have succeeded in improving Transformer-based LMs using KGs. However, most of them unrealistically assume the problem of Entity Linking (EL), i.e. determining which KG concepts are present in the text, is solved upstream. This thesis covers the limitations of handling EL as an upstream task. It goes on to examine the possibility of learning EL jointly with language modeling, and finds that while this is a viable strategy, it does little to decrease the LM's reliance on in-domain text. Lastly, this thesis covers the strategy of using KGs to generate text in order to leverage LMs' linguistic abilities and finds that even naïve implementations of this approach can result in measurable improvements on in-domain language processing.

Titre: Intégration de connaissances expertes dans des modèles neuronaux profonds pour l'adaptation au domaine dans le traitement automatique de la langue Mots clés: Traitement automatique de la langue, Connaissances expertes, Adaptation au domaine, Réseaux de neurones, Apprentissage profond, Transformers Résumé: Les Modèles de Langage (LMs) de pointe sont capables de converser, résumer, traduire, résoudre des problèmes inédits, raisonner, et manipuler des concepts abstraits à un niveau quasi-humain. Cependant, pour acquérir ces capacités, et en particulier pour acquérir une forme de "bon sens" ou des connaissances spécifiques à un domaine, ils requièrent de vastes quantités de texte, qui ne sont pas disponibles pour toutes les langues ou tous les domaines. De surcroît, leurs besoins en puissance de calcul ne sont atteignables que par quelques organisations, limitant leur spécificité ainsi que leur applicabilité aux données sensibles.

Les Graphes de Connaissances (GCs) sont des sources de connaissances structurées qui associent des concepts linguistiques entre eux par le biais de relations sémantiques. Ces graphes sont des sources de connaissances de haute qualité, préexistantes dans une variété de domaines même peu dotés en ressources, et plus denses en informations que du texte. En permettant aux LMs d'exploiter ces structures d'information, ils sont délestés de la responsabilité de mémoriser les informations factuelles, réduisant la quantité de ressources textuelles et calculatoires nécessaires à leur entraînement, et nous permettant de mettre à jour leurs connaissances à moindre coût, élargissant leur cadre d'application et augmentant leur potentiel de démocratisation.

Diverses approches pour l'amélioration de LMs par intégration de GCs ont démontré leur efficacité. Elles reposent cependant sur la supposition rarement vérifiée que le problème de Désambiguïsation d'Entités Nommées (DEN) est résolu en amont. Cette thèse couvre les limitations de cette approche, puis explore l'apprentissage simultané de modélisation de langue et de DEN. Cette démarche s'avère viable mais échoue à réduire considérablement la dépendance du LM sur le texte issu du domaine. Enfin, cette thèse aborde la stratégie de générer du texte à partir de GCs de manière à mieux exploiter les capacités linguistiques des LMs. Il en ressort que même une implémentation naïve de cette approche peut se solder par de considérables progrès en modélisation de langue dans des domaines de spécialité.

"Everything is burning, Emma, because we told it to set fire to everything. This is the workings of the obedient servant. [...] This is the good product." Said [START_REF] Polat | [END_REF] To a bright future for humanity.

List of Tables

-Introduction

. Context & Motivation

In the field of Artificial Intelligence (AI), the most ambitious goal may be to create an agent capable of performing many or all human cognitive tasks at a human or super-human level. This is Artificial General Intelligence (AGI). There are multiple conceptions for how an AGI might be structured, but Natural Language Processing (NLP) holds a privileged role in most of them as the most obvious tool for humans to give instructions to a flexible system, and perhaps the most universal human cognitive task.

The relationship between AGI and language modeling dates back to the inception of the field, with the proposal of The Imitation Game [START_REF] Turing | Computing machinery and intelligence[END_REF], also commonly known as Turing test which posits that the question "Can digital computers mimic human written language well enough to be indistinguishable from true human writing in a conversational setting? " (A) functionally entails the question "Can machines think? " (B). That is, if the answer to A is yes, then the answer to B must also be yes provided "thinking" is not specifically defined in a way that precludes machines from having this ability. This relies on the functionalist argument that the inner workings of an object should have no bearing on the way it is defined, only its influence on its environment should. One interpretation follows from the solipsistic problem of other minds: only one's mind is known to be true for certain; we have no reliable method for rejecting the possibility that the world may be an illusion and that, rather than equals, other minds could be confabulations of our own or automatons mimicking sapience. Consequently, there is no objective basis to discriminate between such automatons and what might be true minds other than one's own, as such a thing cannot be proven to exist.

Furthermore, the idea that fooling a human into thinking they are exchanging with another human requires a form of higher-order reasoning that may be described as "thinking" is not without merit. As language is the most versatile and universal tool we use to communicate ideas, it appears sensible that proficiency in the communication medium would entail proficiency in reasoning about the concepts being handled. In reality, the lack of specifics in terms of allotted time for the test have led to many unambiguously unthinking language models to arguably pass the test, such as Cleverbot [START_REF] Carpenter | Cleverbot[END_REF][START_REF] Aron | Software tricks people into thinking it is human[END_REF]. Consequently, it seems neither consciousness nor the ability to handle a wide range of cognitive tasks are necessary to fool a human into thinking they are conversing with another human in a practical, time-constrained setting.

Despite this failing of the test, the link between mastery of reasoning about abstract concepts, ability to carry out instructions and mastery of natural language remain relevant. As text (often in the form of natural language, but also program source code, mathematical equations, etc.) is the most common representation of inputs and outputs for cognitive tasks and descriptions of non-cognitive processes, it has been my belief since the beginning of my graduate studies that it is critical for an AGI to master this modality, and if no new fundamental breakthroughs are necessary, the first AGI will be largely based on language modeling. This belief has driven and shaped my work thus far.

While this belief may seem to stem from an absurd leap in logic, given a large enough model, the best policy to predict consequences from the description of a process is to implement an internal simulation of the process (in humans, this "simulation" maps to several cognitive processes such as imagination and reasoning ). For instance, the best policy for learning to predict the sum of two numbers is to learn addition. Given examples of physics experiments, the best policy to learn to predict the results is to derive the laws of physics and simulate the experiments. Given the formulation of a cognitive task, the best policy to predict the answer is to simulate the cognitive process. Hence, the best policy for predicting arbitrary text completions, i.e. modeling language, is to create internal representations of the entities involved and their interactions. As universal approximators [START_REF] Hornik | Multilayer feedforward networks are universal approximators[END_REF], it is not unexpected that given enough parameters, data, and time training, the ability to perform this at a human or super-human level could emerge in neural models. It has been proposed however, that language may be insufficient, as true understanding must be grounded in the world, and therefore AGI must be multimodal [START_REF] Huang | Language is not all you need: Aligning perception with language models[END_REF] and/or embodied [START_REF] Kremelberg | Embodiment as a necessary a priori of general intelligence[END_REF].

In the era of neural networks, especially since the advent of the Transformer architecture, the main approach to progress on language modeling has been to increase the amount of parameters in the model, of training data, and of computational resources used. Projects such as ChatGPT [START_REF] Schulman | ChatGPT: Optimizing language models for dialogue[END_REF] have demonstrated that this approach can achieve outstanding results, particularly for language understanding and generating grammatically correct sentences. However, it does not tend to model knowledge well or efficiently, typically having either low certainty on common-sense knowledge, and/or high certainty on untrue facts. This is likely one of the causes of "hallucinations" [START_REF] Ji | Survey of hallucination in natural language generation[END_REF], a phenomenon affecting large transformer-based conversational language models which leads them to confidently disseminate misinformation. An example of this behavior is provided in figure 1.1, in which ChatGPT provides a correct article title which is relevant to the query, but confidently attributes it to incorrect authors. The article title refers to a real article which is relevant [START_REF] Gordon | Experience-dependent plasticity of binocular responses in the primary visual cortex of the mouse[END_REF], but the metadata (authors, date, DOI) are incorrect.

Another significant weakness of this method is its inefficiency in terms of data and computation requirements, particularly at scale. A good basis for scaling comparison is OpenAI's series of GPT models, as they share a basic architecture and training policy, and differ mostly in the number of parameters and training data:

• GPT-1 [START_REF] Radford | Improving language understanding by generative pre-training[END_REF] has approximately 117 million parameters, was trained on 4.5 GB of text (approximately 7000 books, more text than most people read in their entire lives), and required 1.7 × 10 19 FLoating-point OPerations, or 17 exaFLOP1 to train [START_REF] Epoch | Parameter, compute and data trends in machine learning[END_REF][START_REF] Sevilla | Compute trends across three eras of machine learning[END_REF]. It generated mostly grammatically correct text, but often made mistakes and had trouble remaining consistent for more than a few sentences.

• GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] is approximately ten times larger and trained on ten times more data for an approximately hundred times larger training budget (1.5 × 10 21 FLOP), and succeeds in reliably generating a few paragraphs of grammatically correct, consistent text in a variety of contexts. Most significantly, GPT-2 demonstrated the ability to handle a variety of linguistic tasks such as summarization, creative writing, and question answering with carefully constructed prompts which typically include a few examples of the task.

• GPT-3 [START_REF] Brown | Language models are few-shot learners[END_REF] is over a hundred times larger than GPT-2 with 175 billion parameters requiring 800GB of storage, and trained on nearly twenty times more data (950 GB), and was approximately 200 times more expensive to train (3.1 × 10 23 FLOP). GPT-3 can handle significantly larger amounts of text than GPT-2, and was the first model to be able to follow natural language instructions [START_REF] Ouyang | Training language models to follow instructions with human feedback[END_REF]) (e.g. "Extract all place names from the article below:" followed by a news article) and the first general language model to write computer programs from a list of requested features.

• Little is known about the exact technical details of GPT-4 (OpenAI, 2023), but it is known to have at least approximately a trillion parameters, it is trained on a portion of the 19.5PB Common Crawl [START_REF] Elbaz | Common crawl[END_REF] dataset, and required approximately 2.1 × 10 25 FLOP to train according to estimations by Epoch. Given the findings of [START_REF] Kaplan | Scaling laws for neural language models[END_REF] on optimal relative scaling of LM size versus corpus size, we can expect that the corpus size should exceed 3.45 TB. Based on the computation time ratio compared to GPT-3, the corpus size could exceed 11.26 TB. Its most notable improvements over GPT-3 seem to be its general reasoning skills and its world model, as it is able to make more accurate predictions relating to real-world objects in hypothetical situations.

In summary, this mostly quantitative change in size and training data led to a major qualitative change in potential applications and usefulness. These abilities make GPT-3 and its successors unprecedented tools with the potential to assist cognitive tasks and significantly increase productivity in many white-collar professions. However, this naïve approach is highly inefficient in terms of data and computational requirements. This inefficiency has three major drawbacks, detailed below.

Computing at the billion-parameter scale and above is expensive. The cost of training a model the size of GPT-3 is measured in millions of dollars2 , and running it on a server for collaborative use would cost hundreds of thousands per year3 . This is an expense that most organizations cannot afford. Yet, contracting a large technology corporation such as Alphabet or OpenAI to license their language models is often not an admissible option due to privacy concerns. AI corporations have a vested interest in keeping logs of user interactions in order to improve their products, have a poor track record regarding respect of privacy, and wield disproportionate economic (and therefore litigation) power. Potential client corporations dealing with sensitive data such as personal medical data, military strategic information, defendants' personal information in court proceedings and confidential intellectual property therefore cannot fully trust AI corporations to respect the confidentiality of the information. In addition to the privacy concerns, client corporations may also be interested in adapting and fine-tuning language models to their needs with internal additional data and tasks. This use case, in addition to the prohibitive cost, would typically require very large amounts of data and many man-hours to complete due to the low data-efficiency of large language models, adding yet another barrier to adoption.

Large language models may be reaching a scaling limit. As demonstrated by Kaplan et al., language model performance increases (loss decreases) logarithmically with respect to the log of the number of FLOPs, the size of the training corpus, and number of model parameters. Assuming this relationship holds indefinitely and error converges asymptotically to 0, a practical scaling limit nonetheless lies within the available computational power and corpora which will likely fail to meet the exponential demand. While the idea of approaching these upper bounds may seem absurd, as of the writing of this thesis in 2023 and since 2020, a multifactorial global silicon shortage has been causing computer hardware prices to increase dramatically. It is unclear if and when this issue will be resolved, potentially causing stagnation in available computing power, even for AI-focused corporations. Furthermore, the Common Crawl dataset, the largest text dataset available which preserves a large section of the World-Wide Web, may already be utilized nearly to its full useful extent. While it is nearly 20PB large, versions with markup and low quality content filtered out are much smaller, with one of the largest being the 9.7TB multilingual Colossal Clean Crawled Corpus [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-totext transformer[END_REF][START_REF] Dodge | Documenting large webtext corpora: A case study on the colossal clean crawled corpus[END_REF] 4 . This lines up with the amount of text apparently used to train GPT-4. Substantially increasing the amount of training data will likely significantly reduce its quality, or involve acquiring more data through expensive means such as licensing costly content (e.g. Google Books, scientific articles) and digitizing books.

Lastly, further increasing performance on the current language modeling tasks with current architectures may actually be counterproductive. As outlined by [START_REF] Zhuang | Consequences of misaligned AI[END_REF], when utility functions are misspecified with respect to our true goals, as the performance of AI systems increase on their training task, their performance on real-world tasks increases, plateaus, and subsequently decreases as the model learns to exploit the specificities in their proxy utility function. In conversational assistants for instance, providing false but plausible information in response to a difficult request, thereby tricking the user into believing the assistant is helpful, typically has higher utility than not attempting to give useful information, which creates a perverse incentive to deceive. As demonstrated by Perez et al. (2022a), this type of misalignment is already widespread in conversational language models, and as their capabilities increase, these types of failures may render their use actively harmful in subtle ways that may be difficult to detect. It is currently difficult to determine the impact of language model misalignment, and therefore difficult to determine how much further they can be scaled up before they become actively harmful.

Concerns over the environmental impact of the AI industry have also been raised due to energy use for computation. Although lack of traceability makes estimating the global carbon impact of neural network training difficult, we know that large models individually cause substantial carbon emissions [START_REF] Heikkilä | We're getting a better idea of AI's true carbon footprint[END_REF]. For instance, the training of GPT-3 alone has consumed the equivalent of 500 metric tons of CO 2 in terms of power. This is only approximately fifteen billionths of the global emissions, but scaling power consumption by the amount of FLOPs in training, we can estimate the training of GPT4 to have caused 34000 tons of emissions, or a millionth of the world's consumption, without taking into account the cost of running the models as part of the ChatGPT service. As more corporations pursue this type of AI research and continue to scale models up, and as more organizations train similar models, large language models may become a substantial driver of carbon emissions.

In summary, while language models can be made substantially more powerful by scaling up their size and training corpus, soon, we will likely have to turn to other improvement methods to drive progress in the field. Furthermore, large language models are not realistically democretizable due to the monetary and environmental costs of training and running them; they are inefficient learners that have difficulty assimilating facts; and they are prone to having incorrect beliefs and/or deceiving humans in order to avoid missing out on rewards when discussing topics which they know little about. In essence, we would like address these limitations by increasing language models' crystallized intelligence at low computational cost. We decide to take inspiration from the way we deal with this issue in humans: knowledge augmentation using external resources (see Fig. 1.2). We therefore attempt to leverage external, pre-existing sources of knowledge, by integrating them into Transformer-based language models. In this dissertation, I will discuss the various approaches I have undertaken in the pursuit of this objective, as well as my observations and conclusions, and we will consider directions for future research on this and related topics.

. Thesis Statement

Language modeling consists of evaluating how likely any sequence of tokens 5 (typically words, word pieces, or characters) is to appear in language. The typical approach to achieving this is to learn to predict the probability of a token given the tokens in its context on a large corpus. However, as demonstrated by [START_REF] Zipf | The Psycho-Biology of language[END_REF], language is dominated by a few common words, and vocabulary increases logarithmically as the corpus grows. In all natural language corpora, there are therefore many rare words which occur insufficiently often to satisfactorily estimate these probabilities. This is especially true in domain-specific text, which often contains words for highly specific concepts. As we would like to avoid incurring the aforementioned penalties that come with increasing corpus size exponentially, we seek a form of a priori knowledge which provides us with an alternative source of information.

Several candidate sources of information may come to mind, such as:

• Grammar rules which, among other things, sort words into different parts of speech (PoS) and define which constructions are sensible. These rules are generally imperfect and riddled with exceptions, but could conceivably be used to bootstrap sentence-structure understanding and/or as a soft filter to dampen or amplify the probabilities of various tokens depending on their part of speech, identified with rule-based dependency parsing.

• Weaker language models used in a reverse-distillation setting, i.e. before self-supervision on text, a larger, newer model is pre-trained on the outputs of an older, smaller model, which creates lower-quality but more informative data.

• Dictionaries and encyclopediae, which can be added to most corpora and trained on for multiple epochs 6 or queried to add context during inference.

• Ontologies and Knowledge Graphs 7 (KGs) which arrange concepts in a web of relations which capture probabilistic dependencies between concepts (and, therefore, tokens).

5 Token: "A token is an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing." (Manning et al., 2008, p. 22) 6 Epoch: complete pass over the linguistic resource 7 Ontologies are a type of Knowledge Base (KB), but not all Knowledge Bases are technically Ontologies. Knowledge Graphs are a common way to represent an Ontology, but not necessarily the only way. In practice however, these nuances are not critical. Henceforth, we will generally be using the terms 'Knowledge Base', 'Ontology', and 'Knowledge Graph' interchangeably to mean "an ontology which is represented as a knowledge graph for our practical purposes". We tend to prefer 'Knowledge Base' when speaking in general terms and 'Knowledge Graph' when discussing the technical details of an approach which is specific to graphs.

Knowledge graphs are of particular interest as an under-utilized source of highquality information. Experts in many fields have carefully constructed such knowledge graphs as WorldKG [START_REF] Dsouza | Worldkg: A world-scale geographic knowledge graph[END_REF], a geographically-oriented KG; WordNet [START_REF] Miller | Wordnet: a lexical database for english[END_REF], a general knowledge graph representing synonymy and other lexical relations between words; and the Unified Medical Language System (UMLS), which indexes almost all biomedical concepts, along with their various spellings and different types of mutual relationships.

In this dissertation, we explore the various ways in which we might leverage knowledge graphs in support of Transformer-based language models for the purposes of alleviating the previously stated drawbacks of the naïve approach. Specifically, we will be using UMLS as a basis for our work: while using multiple knowledge graphs in the exploration of our methods for leveraging them would likely yield more nuanced results and a deeper understanding, adapting our methods to each specific KG would cost a significant amount of time which we believe is better spent broadening the scope of our exploration to multiple knowledge integration techniques. We specifically chose UMLS based on the set of challenges that it does and does not pose:

• Lack of text is its own difficult problem to tackle in the context of Transformerbased language models. It is therefore desirable to work within a field where this is not an issue as, if a low-resource scenario is relevant to explore in any given work, this can still be simulated in a non-resource-constrained domain.

The biomedical domain has no shortage of textual resources, including millions of Open Access articles (as shown in Figure 1.3), making it a good candidate field to focus on.

• UMLS specifically is the most extensive KG in the biomedical domain, not only affording the most options, but also being large enough to be challenging to manage. Systematic text-based term searches are not tractable at the scale of language model training corpora, requiring specific optimization efforts which lead our research in a productive direction. This also ensures our methods are likely to be tractable on almost all other knowledge graphs, as there are very few knowledge graphs which are comparable in size to UMLS.

• Although this is subjective, we find working on the biomedical domain helps envision applications for our work which may help patients or researchers, and find these prospects particularly inspiring.

Despite our exclusive use of UMLS and focus on the biomedical domain however, the concepts discussed in this thesis are meant to apply to any Ontology and any domain. In summary, we will explore the integration of knowledge from knowledge graphs into Transformer-based LMs for the purpose of adapting LMs to specific domains while reducing the computational (and non-computational) costs in comparison to the naïve approach. We will closely examine three major approaches in particular, report our findings, and discuss future directions for research on knowledge integration, domain adaptation, and data efficiency in language modeling.

. Outline

In the following chapter, we will review the literature on the topics of domain adaptation in language modeling and knowledge integration, as well as the prerequisites in the various neural approaches to language modeling and the task of Entity Linking, which is a pivotal task for many approaches to knowledge integration. Chapters 3 through 5 will be dedicated to my original contributions to the field. First will be discussed the importance of Entity Linking (EL) as a precursor to knowledge integration methods such as ERNIE [START_REF] Zhang | ERNIE: Enhanced language representation with informative entities[END_REF], UmlsBERT [START_REF] Michalopoulos | UmlsBERT: Clinical domain knowledge augmentation of contextual embeddings using the Unified Medical Language System Metathesaurus[END_REF] or BERT-MK [START_REF] He | Integrating graph contextualized knowledge into pre-trained language models[END_REF], which are based on the identification of entities mentioned in the text, and the extraction of corresponding entity embeddings. In chapter 4, we will examine ways to work around the Entity Linking problem within the entity-embedding-based methods, in particular applying the KnowBert [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF] method, which introduces the idea of fuzzy entity linking, to the biomedical domain with UMLS. We will discuss the inherent limitations of this type of approach, and in chapter 5 will consider alternative classes of approaches and explore data augmentation through text generation as a specific instance.

Lastly, chapter 6 will conclude by summarizing these contributions; discussing future avenues for investigation including potential improvements and different approaches to knowledge integration; and reexamining the current direction of Machine Learning and Language Modeling research, the problems faced in the field, and the potential roles of knowledge integration and domain adaptation in that future.

-Background & Related Work

. Language Modeling

Computational language modeling traces its roots to the work of al Kindi (9th century) on the statistical study of language in cryptography. National defense and Intelligence were the primary drivers of progress in what would eventually be known as Natural Language Processing during the mid-twentieth century, with applications in cryptography during the war, most notably Alan Turing's work at Bletchley Park, and in Machine Translation (MT) afterwards and into the early 1960s (Weaver, 1949(Weaver, /1955)), until the publication of the unfavorable ALPAC report [START_REF] Pierce | Returns to investment in higher education[END_REF]. Influenced by Turing's idea of the imitation game [START_REF] Turing | Computing machinery and intelligence[END_REF] as the litmus test for true machine intelligence, the development of the conversational Rogerian psychotherapist chatbot ELIZA [START_REF] Weizenbaum | Eliza-a computer program for the study of natural language communication between man and machine[END_REF] marks a pivotal event in applied computational linguistics, bridging the gap with Chomskyist theory [START_REF] Chomsky | Syntactic structures[END_REF]. Since the 1970s, many ontologies and KBs have been developed as, among other applications, a supporting framework to expand the range of capabilities of symbolic NLP systems influenced by ELIZA such as MARGIE [START_REF] Schank | Margie: Memory analysis response generation, and inference on english[END_REF] and TALE-SPIN [START_REF] Meehan | TALE-SPIN, an interactive program that writes stories[END_REF]. Many of these knowledge bases contain valuable information, distilled to a relatively dense representation, and some remain in use. A notable example of this is the Unified Medical Language System (UMLS) [START_REF] Lindberg | The unified medical language system[END_REF], of which we have made ample use in the context of this research.

Starting in the 1970s, with the introduction of the concept of inverse document frequency (IDF) in particular [START_REF] Jones | Index term weighting[END_REF], statistical approaches to computational linguistics have gained traction. As an increasing number of inconsistencies within and across natural languages were found, rule-based models have become increasingly complex. The apparent increasing complexity of the hypothetical set of rules defining the universal grammar posited by Chomsky as the underlying implementation of the human language acquisition device, in addition to the continuous improvement of computer hardware, led to the dominance of these statistical approaches in the 1990s. Improvements to statistical tools for text mining and information retrieval such as LSA [START_REF] Deerwester | Indexing by latent semantic analysis[END_REF] and a breakthrough in corpus-based machine translation spearheaded by IBM with their alignment models [START_REF] Brown | The mathematics of statistical machine translation: Parameter estimation[END_REF] in particular laid the foundation for modern statistical NLP. This coincided with development of the modern World Wide Web, which provided both an increasingly large amount of easily harvested text, enabling easy statistical study of language, as well as an incentive to develop these tools in order to organize, index, search, and study the web and its users.

While neural networks were long criticized for their lack of interpretability and computational cost, interest in their applications to text increased throughout the 2000s due in large part to the work of Bengio and Hinton in neural language modeling [START_REF] Bengio | A neural probabilistic language model[END_REF][START_REF] Morin | Hierarchical probabilistic neural network language model[END_REF][START_REF] Mnih | A scalable hierarchical distributed language model[END_REF].

Graphics processing hardware and firmware improvements in the late 2000s and early 2010s dramatically increased the time efficiency of training neural networks, and led to increased interest by the machine learning community. The application of Recurrent Neural Networks (RNNs) to language modeling [START_REF] Mikolov | Recurrent neural network based language model[END_REF] and subsequent advancements in neural word embeddings attributable to Word2Vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] and GloVe [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] marked a turning point in neural NLP leading to significant progress in all sub-fields, perhaps most notably in sequence-to-sequence tasks such as Neural Machine Translation (NMT) (Bahdanau et al., 2015a) and syntactic parsing [START_REF] Vinyals | Grammar as a foreign language[END_REF] in the case of RNN models and information retrieval and classification tasks (e.g. sentiment analysis, topic detection, word and document similarity in the case of word embeddings.

Despite the improvements of pretrained neural word embeddings, the major remaining limitations were the limited vocabulary and 1-to-1 mapping of vectors to words in that vocabulary leading to limited ability to model word polysemy. In the late 2010s, [START_REF] Peters | Deep contextualized word representations[END_REF] addressed these issues using an RNN-based LM, ELMo, to contextualize distributed word representations using a characterbased input. As a result, any character sequence, even if it had never been seen in training, could generate an embedding which took context into account and thus differentiated between the possible meanings of a word.

Another independently-developed major addition to language modeling in the late 2010s was the use of the self-attention mechanism to allow models to assign various levels of importance to various input tokens depending on the context. Rooted in the work of [START_REF] Schmidhuber | Learning to control fast-weight memories: An alternative to dynamic recurrent networks[END_REF] on fast weights in neural networks (weights which are variable at inference), attention mechanisms were popularized in the context of language modeling by Bahdanau et al. (2015a) and entirely replaced recurrent architectures in the wake of the publication of Attention Is All you Need by [START_REF] Vaswani | Attention is all you need[END_REF]. We discuss the new, fixed-context-size non-recurrent attention-based architectures, called Transformers, in Section 2.3.

. Recurrent Neural Networks

Whilst in theory, simple Multilayer Perceptrons (MLPs) with fixed inputs and outputs can approximate any function [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF], they are not capable of handling variable-size inputs such as images, sound, data streams, time series, and text. This problem has been addressed in multiple different ways depending on the application. Scaling (using an algorithm such as nearest-neighbor or bicubic interpolation) and cropping are common approaches in computer vision, while sliding context windows and input padding are common approaches for most other applications. These methods have the drawback of corrupting the input in some way and are limited in the amount of context that can be drawn from. The Recurrent Neural Network (RNN) architecture, based on the work of [START_REF] Rumelhart | Learning internal representations by error propagation[END_REF] and [START_REF] Robinson | The utility driven dynamic error propagation network[END_REF] but proposed in its most common form by [START_REF] Elman | Finding structure in time[END_REF], is an alternative solution to this problem, allowing for sequential data to be fed ad libitum into a network over the course of multiple steps, during which the network maintains and updates a state vector which memorizes the useful features of the data which has been supplied up to that point.

Many RNN architectures exist, which share the basic trait that the neuron graph contains cycles. The Elman network architecture is the most common and simply includes one hidden layer connected to itself. To resolve the backwards connections, they are considered an output at each time step t, and an input at each time step t + 1 as shown in Fig. 2.1. Formally, the output o t and hidden layer h t are expressed as:

o t = σ(W o h t + b o ) h t = σ(W h h t-1 + W x x t + b h )
Where x t is the input at step t; matrices W and vectors b are (respectively) connection weights and biases to be learned; and σ is the non-linear activation function (traditionally sigmoid). h 0 is typically initialized to the ⃗ 0 vector. The Recurrent Neural Network architecture can theoretically encode information over any length of context1 (and is in fact Turing-complete ;[START_REF] Hyötyniemi | Turing machines are recurrent neural networks[END_REF], limited in the resolution of the information only by the size of h. However, in its classical form, this architecture is met with difficulty scaling to large numbers of inputs as shown by [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF] and [START_REF] Pascanu | On the difficulty of training recurrent neural networks[END_REF]. In practical NLP, this means RNN-based language models have trouble scaling to multiple sentences. This is fundamentally due to a property of chaining multiplications, that is, that repeated multiplications of numbers whose absolute value is smaller or larger than 1 will tend to 0 or ±∞ respectively as the number of operations increases, such as is the case with gradient Backpropagation Through Time (BPTT) and iterated forward propagation of inputs. This has two consequences. The first is that as the length of the context grows, gradient descent ceases to be effective at updating the weights of the network, as the gradient approaches 0 ("vanishes") or diverges ("explodes"). The second is that, as there is no mechanism allowing for an explicit, binary decision to be made regarding whether to conserve or discard a piece of information, the slightest degradations to the information will inevitably compound as context size increases. The Long Short-Term Memory (LSTM) network architecture introduced by [START_REF] Hochreiter | Long short-term memory[END_REF], as well as the architectures it has inspired such as the Gated Recurrent Units (GRU) [START_REF] Cho | On the properties of neural machine translation: Encoder-decoder approaches[END_REF] attempt to address these limitations by introducing various sub-networks, or "gates", which explicitly control the information that should be retained and lost, and cause the gradient to be computed in such a way that it is less likely to vanish or explode.

As shown in Fig. 2.2, the LSTM unifies the hidden state vector h with the output, and adds to it a recurrent explicit context memory vector c, which is updated with a candidate information vector z, a memorization gate i (which rescales the components of the candidate vector z), and a forget gate f (which rescales the components of the memory vector c). Additionally, an output gate o is trained to decide which parts of the memory vector to incorporate into the output. Formally, the forward pass is computed thusly:

h t = f (c t ) ⊙ o t c t = c t-1 ⊙ f t + z t ⊙ i t o t = σ(U o h t + W o x t + b o ) z t = tanh(U c h t + W c x t + b c ) f t = σ(U f h t + W f x t + b f ) i t = σ(U i h t + W i x t + b i ) (2.1)
Where ⊙ is the element-wise product, and f is usually tanh, constraining the output to the [-1, 1] range, but can be replaced by the identity function.

Thanks in particular to the reliance on addition rather than multiplication for updates to the memory vector, this architecture minimizes the number of successive multiplications in the gradient, reducing its risk of exploding or vanishing. LSTM networks and related gating mechanisms thus prove to be effective at increasing the allowable context size of RNNs. However, this comes at a significant computational cost due to the increased number of parameters to tune.

Additionally, since long-term dependencies must be learned over a number of steps proportional to the distance between terms, these models typically fail to learn to handle context sizes in excess of a few paragraphs of text, which is highly restrictive in terms of the scope of the types of text that can be processed. Another shortcoming of the RNN architecture resides in its inherently serial structure, i.e. the result of time step t must be known to compute the result of time step t + 1. Therefore, processing and learning over long sequences, even where possible, is not highly parallelizable.

. Transformers

Hi

Scaled dot-product attention

Linear Linear Linear

Scaled dot-product attention Linear Linear Linear 2.3) to encode input text from a source language and Transformer Decoder blocks (Fig. 2.4) to generate a sentence in the target language, informed by the representation given by the encoders. The decoder block is almost identical to the encoder block but includes a Masked Multi-Head Self-Attention module before the Multi-Head Self-Attention module, and the Multi-Head Self-Attention module accepts the output of the Encoder blocks as input. The Masked Multi-Head Self-Attention module differs from the (nonmasked) Multi-Head Self-Attention module in the fact that it prevents the attention mechanism from propagating information backwards in the sequence -for each word, information from its predecessors can be attended to, but its successors are "masked". The original model uses 6 encoder blocks, 6 decoder blocks and h = 8 attention heads. This architecture works well for translation and similar tasks, but is purely a sequenceto-sequence architecture and is fairly complex, requiring many parameters. The Transformer uses byte-pair encoding [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF], which works around the problem of out-of-vocabulary words and is resilient to typos in input text without limiting context size as much as characterbased tokenization by representing common character combinations as single tokens. Subsequent Transformer-based architectures tend to use the similar WordPiece tokenization [START_REF] Wu | Google's neural machine translation system: Bridging the gap between human and machine translation[END_REF].

1 √ d k • Softmax Q K V Linear
ELMo [START_REF] Peters | Deep contextualized word representations[END_REF], though it was based on the LSTM architecture, introduced the concept of contextualized embeddings. That is, rather than assigning a fixed vector to every word in a predetermined vocabulary as per Word2Vec, each token in a sequence can be assigned a vector which accounts for its general use and its local context. Therefore, polysemic words can be differentiated.

This approach can also detect when a word is used in a novel context which has not been seen in training. This was achieved by training the output representations to optimize for the autoregressive language modeling objective, which consists of predicting the next word in a sequence. To be more specific, ELMo does this bidirectionally using a bidirectional LSTM. Optimizing for this objective yields token representations which model grammatical and semantic information for that token given its context, which can then be used for a variety of downstream tasks such as Named Entity Recognition (NER) (i.e. tagging words in a sequence which mention an entity of interest and typically assigning it a type such as "location" or "person"), Part of Speech (PoS) tagging (i.e. tagging words based on their grammatical function in the text), or Natural Language Inference (i.e. recognizing the type of relation between two sequences such as "contradiction" or "entailment"). [START_REF] Peters | Deep contextualized word representations[END_REF] further introduce the idea of fine-tuning the language model for a specific task. This approach is further expanded upon and standardized by the ULMFiT approach [START_REF] Howard | Universal language model fine-tuning for text classification[END_REF].

GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF] eschews the encoded context sequence used by Vaswani et al.'s translation model and adopts a purely generative architecture based on the Transformer Decoder block (Fig. 2.3) which is optimized for unidirectional, autoregressive language modeling (i.e. predicting the next word in a sequence). As it does not integrate Encoder output, it also simplifies the Decoder block by removing the Multi-Head Self-Attention module. This yields a model that is particularly well-suited to generative tasks, such as conversational agents and creative writing. Crucially, as shown by GPT-2 and Instruct-GPT [START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Ouyang | Training language models to follow instructions with human feedback[END_REF], many tasks can be formulated as generative and/or conversational tasks, such as summarization and translation. With 12 transformer decoder blocks and 12 attention heads, GPT comprises approximately 117 million parameters.

BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] adapts the ELMo approach of word embedding contextualization to Transformers. It takes the opposite approach to GPT and uses only Transformer Encoder blocks (Fig. 2.3) to derive contextualized distributed word representations. It adapts ELMo's bidirectional objective to the transformer by introducing the Masked Language Modeling (MLM) objective: words in the input sequence are masked with some probability (15% in the case of BERT), and the model must learn to predict which words were masked. In addition to these adaptations, BERT includes a special token called [CLS] which models the entire sequence for sequence classification purposes. Lastly, [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] introduce the Next Sentence Prediction (NSP) objective which consists of feeding pairs of sequences to the model and teaching it to differentiate between sentences which are contiguous in the original text from sentences which are unrelated to each other. BERT is particularly effective in the same contexts as ELMo, i.e. Natural Language Understanding (NLU) tasks (which typically involve classification). However, while not as effective at creating coherent text as the GPT approach, by masking the last word(s) in a sequence, BERT can be used as a generative language model. With 12 transformer encoder blocks and 12 attention heads (the same as GPT), BERT BASE comprises approximately 110 million parameters. [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] also released a 24-encoder 16-head 340 million parameter model dubbed BERT LARGE . When not specified, "BERT" generally refers to BERT BASE .

Input text in transformers is generally represented as a sum of multiple embeddings for each token:

• Each token in the vocabulary is associated with a token embedding which may be pretrained or learned as part of the language model training. • Position embeddings are token-invariant embeddings assigned to the positions in the input, generally based on a mixture of sine waves at different frequencies.

• In models which may take multiple input sequences such as BERT, sequence embeddings (which are generally trained in tandem with the LM objective) may be added to differentiate them.

These are the most standard types of input embedding components, but some approaches such as UmlsBERT [START_REF] Michalopoulos | UmlsBERT: Clinical domain knowledge augmentation of contextual embeddings using the Unified Medical Language System Metathesaurus[END_REF] may introduce additional components.

BERT is highly effective in NLU tasks and being a small enough model to study for most organizations, the barrier to entry is low in studying BERT-based architectures, making it a prime choice for many NLP research applications. GPTbased architectures only display high enough performance to be useful at the scale of GPT-2, which as approximately 1.5 billion parameters -approximately 14 times larger than BERT BASE . Many improvements have thus been made to BERT in the NLP literature. Notably, it has been found that in practice, the NSP objective is ineffective, or even counter-productive [START_REF] Yang | Xlnet: generalized autoregressive pretraining for language understanding[END_REF][START_REF] Liu | Roberta: A robustly optimized BERT pretraining approach[END_REF][START_REF] Lan | ALBERT: A Lite BERT for self-supervised learning of language representations[END_REF]. Notable contributions to the BERT literature (sometimes referred to as "BERTology" [START_REF] Rogers | A primer in BERTology: What we know about how BERT works[END_REF]) include the findings of [START_REF] Liu | Roberta: A robustly optimized BERT pretraining approach[END_REF] on transformer hyperparameterization and the effects of corpus size, [START_REF] Yang | Xlnet: generalized autoregressive pretraining for language understanding[END_REF]'s autoregressive extension of BERT, and improvements on BERT's scaling by [START_REF] Lan | ALBERT: A Lite BERT for self-supervised learning of language representations[END_REF] (particularly in terms of memory requirements). For further review and examples of landmark Transformer-based language models in the context of the encoder-based and decoder-based families, readers are directed to the work of [START_REF] Yang | Harnessing the power of LLMs in practice: A survey on ChatGPT and beyond[END_REF].

. Domain Adaptation

In the wake of the advent of the Transformer architecture introduced by [START_REF] Vaswani | Attention is all you need[END_REF], language processing tasks have increasingly been handled by neural language models based upon this architecture such as GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF] and BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Many projects have sought to leverage the Transformer architecture in specific fields ; for instance in the process of discovery, in the legal domain [START_REF] Yang | Goldilocks: Just-right tuning of BERT for technology-assisted review[END_REF], which has traditionally required lawyers to sift through large numbers of documents, or patient risk modeling for disease prognosis in the medical domain [START_REF] Li | Hi-behrt: Hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records[END_REF]. Due to the distributional shift between the language used in specific fields and the types of text typically used to train these models however, general language models tend to underperform in these fields [START_REF] Yang | Goldilocks: Just-right tuning of BERT for technology-assisted review[END_REF][START_REF] Lee | BioBERT: a pretrained biomedical language representation model for biomedical text mining[END_REF].

. Definition

Most definitions of domain adaptation, such as the one provided by [START_REF] Ruder | Neural transfer learning for natural language processing[END_REF], describe it as a type of transfer learning characterized by the source and target settings of the model differing in data distribution rather than task, as opposed to fine-tuning, which focuses on adapting a model to a new task. For instance, in computer vision, training a model to categorize dogs and cats before retraining it to categorize cancerous versus non-cancerous cells would be an example of domain adaptation, whereas training a model to reconstruct corrupted images (a task which seeks to teach general features of images) before changing the model objective to classification would be an example of pretraining and fine-tuning.

However, on one hand, transfer learning fundamentally seeks to retrain an existing model to perform well on a certain task for a certain distribution of data which may differ from the task and data for which the original model was optimized. On the other, domain adaptation seeks simply to efficiently re-use a model, which was trained on a given task and source distribution, with a new target distribution. The latter does not inherently involve additional learning and therefore does not entail the former. Consequently it is arguably inaccurate to describe domain adaptation as a type of transfer learning in theory, despite most practical implementations of domain adaptation also being instances of transfer learning. We thus broaden the scope of domain adaptation and define it thus: Domain Adaptation is any process by which an existing model, optimized for a source data distribution, is altered so that it may better perform on a target data distribution.

Desirable characteristics for domain adaptation techniques include:

• Increased data efficiency in comparison with training the model from scratch on the target domain

• Increased computational efficiency in comparison with training the model from scratch on the target domain

• Applicability to models not specifically intended for domain adaptation

• Non-necessity of taking target domain into account for the creation of the source-domain model

• Reverse-compatibility with the source distribution 2.4.2 . Historical approaches In its most basic form, domain adaptation consists of correcting a biased statistical classifier trained on N s sample-label pairs (x, y) drawn from the source joint distribution D s (X, Y ) and deployed on data from the target joint distribution D t (X, Y ). It may also be trained on N t pairs drawn from the target joint distribution D t (X, Y ) with 0 < N t ≪ N s . This is a fundamental problem in statistics [START_REF] Shimodaira | Improving predictive inference under covariate shift by weighting the log-likelihood function[END_REF] and a classic problem in econometrics [START_REF] Heckman | Sample selection bias as a specification error[END_REF], [START_REF] Zadrozny | Learning and evaluating classifiers under sample selection bias[END_REF]), but has been studied under the name of domain adaptation as a machine learning problem since the mid-2000s with the work of [START_REF] Daumé | Domain adaptation for statistical classifiers[END_REF]; [START_REF] Blitzer | Domain adaptation with structural correspondence learning[END_REF]; and [START_REF] Jiang | Instance weighting for domain adaptation in nlp[END_REF].

The classical approach to solving this problem as described by [START_REF] Jiang | A literature survey on domain adaptation of statistical classifiers[END_REF] is to insert a bias correction term in the training of the classifier. The optimal set of parameters θ * for the model is obtained by the following equation: argmin is an optimization algorithm such as SGD. While P s (x i , y i ) can be estimated using the observed sample Ps (x i , y i ) from the source distribution, estimating λ ← Pt(x i ,y i ) Ps(x i ,y i ) is not straightforward because (x i , y i ) are from the source domain and the target distribution sample Pt is small. Many domain adaptation techniques consist of finding satisfactory estimates for λ.

θ * t =
Class Imbalance: Under this special case, we assume the conditional distributions of the samples given the labels are the same in both domains, but the marginal distributions of the labels may differ. This ratio of label occurrences can typically be satisfactorily estimated using λ ≈ Pt(y) Ps(y) [START_REF] Lin | Support vector machines for classification in nonstandard situations[END_REF]. This is roughly equivalent to resampling the source distribution in order to match the target label distribution.

I.e. D s (X|Y = y) = D t (X|Y = y) but D s (Y ) ̸ = D t (Y ).
Covariate Shift: This special case is the reverse of the former, in which we assume the conditional distributions of the labels given the samples are the same in both domains, but the marginal distributions of the samples may differ.

I.e. D s (Y |X = x) = D t (Y |X = x) but D s (X) ̸ = D t (X).
In traditional single-step training procedures (as opposed to multi-objective and pretrain/fine-tune procedures), this problem is only relevant if the model cannot perfectly fit the entire distribution and must fit the dense regions of X preferentially [START_REF] Shimodaira | Improving predictive inference under covariate shift by weighting the log-likelihood function[END_REF]. Following the same bayesian substitution process, we derive:

P t (x, y) P s (x, y) = P t (x) P s (x)
(2.7)

As each x typically appears rarely in the dataset, estimating Pt(x) Ps(x) is not straightforward. Multiple methods have been proposed such as adding λ as a learned parameter of the model [START_REF] Bickel | Discriminative learning for differing training and test distributions[END_REF] and non-parametric kernel density estimation [START_REF] Shimodaira | Improving predictive inference under covariate shift by weighting the log-likelihood function[END_REF].

For a more in-depth overview of this class of methods, we refer readers to the work of [START_REF] Jiang | A literature survey on domain adaptation of statistical classifiers[END_REF]. However, these approaches are not typically applicable in the context of language modeling. To illustrate this, let us take the basic case of autoregressive language modeling, which models the probability of each token as being conditioned by its preceding context (i.e. left-hand context in left-to-right languages). In this case, X is the set of all contexts and Y is the set of all tokens. In theory, context may be infinite, but in practice is bounded -however, larger contexts are preferable as they allow for the modeling of long-range dependencies. As context size increases, the number of possible contexts increases, P t (x) and P s (x) converge towards 0, and Pt(x,y) Ps(x,y) is thus undefined. Other methods include the proposal by [START_REF] Daumé | Frustratingly easy domain adaptation[END_REF] to learn a mapping from a k-dimensional input space to a (n+1)k-dimensional input space where n is the number of domains. In essence, this approach learns domain-specific functions for words, e.g. the word short would likely be predominantly used as an adjective in the general domain, but a verb in the domain of economics.

Maximum Mean Discrepancy (MMD) is a method for estimating the difference between two distributions from a set of samples, defined as the difference between distributions in the reproducing kernel Hilbert space [START_REF] Gretton | A kernel method for the two-sample-problem[END_REF]. This has been used as a basis for transfer component analysis [START_REF] Pan | Domain adaptation via transfer component analysis[END_REF], which projects sample representations into a lower-dimensional space which minimizes MMD, and as a loss for domain regularization in order for the model to learn domain-invariant latent representations [START_REF] Ma | Domain adaptation with BERT-based domain classification and data selection[END_REF]. However, MMD by construction reduces feature discriminability (Wang et al., 2021a).

Grassmann Manifolds have been used by [START_REF] Gong | Geodesic flow kernel for unsupervised domain adaptation[END_REF] and [START_REF] Gopalan | Domain adaptation for object recognition: An unsupervised approach[END_REF] to find domain-invariant representations. This is done by embedding the domains in lower-dimensional subspaces, constructing a geodesic curve in the Grassmannian, and either sampling subspaces from that curve in order to perform projections of the data and construct a unified embedding through concatenation [START_REF] Gopalan | Domain adaptation for object recognition: An unsupervised approach[END_REF] or using this geodesic to compute a kernel (subsequently used to build a kernel-based classifier) [START_REF] Gong | Geodesic flow kernel for unsupervised domain adaptation[END_REF].

Autoencoders are neural networks which are trained to output their own input (or an uncorrupted version of a corrupted input) but contain a small hidden layer so as to create a representation bottleneck forcing the model to learn to compress the input. They have been used in a similar fashion to Grassmann manifolds [START_REF] Glorot | Domain adaptation for large-scale sentiment classification: A deep learning approach[END_REF][START_REF] Chen | Marginalized denoising autoencoders for domain adaptation[END_REF]: the objective remains to find a smaller, domain-invariant representation of the input. Neural Structural Correspondence Learning (NSCL) was introduced by Ziser and Reichart (2017) as a variation on the autoencoder approach which adds support for an explicit mechanism for learning to leverage both cross-domain and domain-specific features in tandem. In practice however, [START_REF] Lin | Does BERT need domain adaptation for clinical negation detection[END_REF] do not find NSCL to be helpful in the context of Transformer-based LMs.

Approaches based on minimizing MMD, Grassmann manifolds or autoencoders, which seek to find domain-invariant representations, display a significant limitation, as they essentially consist of finding the general features which apply across domains. This can be seen as a compromise between source and target domain, improving performance on the target domain at the cost of source-domain performance by neglecting source-domain-specific features. This performance reduction, in traditional single-stage model training curricula, is simply underfitting, whereas in more complex settings such as the pretrain/fine-tune paradigm, this problem is referred to as catastrophic forgetting. In an ideal scenario, we would prefer to be able to leverage the specificities of both domains. This is by definition easy (and in fact the default behavior) in the source domain, but inherently requires, if not data, some a priori knowledge of the target domain. Additionally, these techniques require knowledge of the target domain before the model is optimized for the source domain, reducing their scope of applicability.

Another major drawback of these domain adaptation methods is that they must be applied for each individual sub-task, with most of them being restrained to classification tasks such as part of speech tagging or named entity recognition in NLP. In a pretrain/fine-tune paradigm, these methods would therefore have to be applied at the fine-tuning stage, and would require sets of labeled data for both the source and target domain within each task, multiplying the need for annotated data and further narrowing their range of relevancy. In addition, these methods are limited to leveraging the information in the source domain and a small amount of information from the target domain to perform domain adaptation. In Natural Language Processing, this may not be sufficient as relationships between entities can often only be learned from resources in the target domain; e.g. information on the relationship between pancreatic β-cells, insulin production and diabetes is unlikely to occur in general-domain text, and no corrective factor or other mathematical trick could induce modeling of these relationships by the language model if the information is not presented in some form. We therefore seek to make use of additional sources of information while minimizing the negative effects of increasing corpus size.

. In computer vision

Domain adaptation for deep neural networks was largely popularized by the success of transfer learning in computer vision [START_REF] Yosinski | How transferable are features in deep neural networks? Advances in neural information processing systems[END_REF][START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF], where fine-tuning image classification models trained on natural image datasets such as ImageNet [START_REF] Deng | Imagenet: A largescale hierarchical image database[END_REF] yielded highly superior results to models trained from scratch on domain-specific image datasets, which often were comparatively small. The most notable examples of this were in the biomedical domain [START_REF] Greenspan | Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique[END_REF], and particularly for detection of cancerous growths [START_REF] Esteva | Dermatologist-level classification of skin cancer with deep neural networks[END_REF] and lung disease [START_REF] Shin | Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning[END_REF], where human or arguably super-human level has been reached [START_REF] Rajpurkar | Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning[END_REF][START_REF] Goh | Evaluating human versus machine learning performance in classifying research abstracts[END_REF].

. In-domain extended pre-training in Transformers

Transformer-based language models, when trained mostly on general text such as is the case with BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] and GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF], for which the bulk of the training corpus is formed by the Books corpus [START_REF] Zhu | Aligning books and movies: Towards story-like visual explanations by watching movies and reading books[END_REF], do not perform well on tasks involving specific domains such as medicine, patents, or law. This is to be expected, as many of the writing conventions differ by field and make assumptions on the knowledge of the reader. The obvious way to expand the capabilities of a language model to a specific domain is therefore to include in-domain text in the model's training corpus. This has indeed been the typical approach, and has been quite successful, with its performance having been investigated by [START_REF] Gururangan | Don't stop pretraining: Adapt language models to domains and tasks[END_REF], and with notable models such as GeoBERT [START_REF] Gao | Geobert: Pre-training geospatial representation learning on point-of-interest[END_REF], LEGAL-BERT [START_REF] Chalkidis | LEGAL-BERT: The Muppets straight out of law school[END_REF], or PatentBERT [START_REF] Lee | Patent classification by fine-tuning BERT language model[END_REF]. The biomedical domain in particular has inspired a variety of models which all incorporate various amounts and proportions of biomedical text in their pre-training phase, e.g. BioBERT [START_REF] Lee | BioBERT: a pretrained biomedical language representation model for biomedical text mining[END_REF], BlueBERT [START_REF] Peng | Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets[END_REF], BioMed-RoBERTa [START_REF] Gururangan | Don't stop pretraining: Adapt language models to domains and tasks[END_REF], SCIBERT [START_REF] Beltagy | Scibert: A pretrained language model for scientific text[END_REF], and ClinicalBERT [START_REF] Alsentzer | Publicly available clinical BERT embeddings[END_REF].

However, not only is this type of pre-training sensitive to catastrophic forgetting -i.e. it usually leads models to underperform on general-domain text as demonstrated by [START_REF] Arumae | CALM: Continuous Adaptive Learning for Language Modeling[END_REF] and [START_REF] Xu | Forget me not: Reducing catastrophic forgetting for domain adaptation in reading comprehension[END_REF]; large models with attention mechanisms such as BERT are also notoriously computationally expensive to pre-train. Furthermore, the ability for a model to associate concepts (e.g. "COVID-19 " and "respiratory failure") is predicated on these concepts appearing in the pre-training corpus, leading to difficulty adapting to some forms of distributional shift. These limitations have lead to interest in different methods of adapting these models to specific domains.

. Catastrophic Forgetting

As models trained on one task and data distribution are repurposed for a new task (transfer learning) and/or a new distribution (domain adaptation), the finetuning process has no incentive to allow the model to continue performing well on the source task/domain, and critical parameters can be modified. This not only lowers performance of the model on the source task/domain (which may be problematic for large models which are expensive to run or host as a service), but also may lead optimizers such as Stochastic Gradient Descent (SGD) or Adam to converge to poor local optima on the target task/distribution. This problem, which we've touched on as a weakness of some existing domain adaptation techniques, is known as Catastrophic Forgetting (CF).

Experience Replay [START_REF] Isele | Selective experience replay for lifelong learning[END_REF][START_REF] Arumae | CALM: Continuous Adaptive Learning for Language Modeling[END_REF]) may be the most obvious way to address CF. Experience Replay is a type of Multi-task learning which specifically re-uses the training data from the source task in order to alternate source and target task and/or distribution in the model's training curriculum. This provides the model with the missing incentive to conserve performance on the source task/distribution, but as the objectives are alternated rather than united, the Experience Replay mechanism by construction moreso undoes the damage done by the fine-tuning process than protects previously acquired knowledge, creating an inefficient optimization path in the loss landscape. Combined with the inherent additional computation of replaying past data, it is a computationally expensive way to counter CF.

Adapters [START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF][START_REF] Pfeiffer | Adapterhub: A framework for adapting transformers[END_REF] mitigate CF by freezing all model parameters after pretraining and adding small layers of new parameters within Transformer blocks which take on the responsibility of fitting to the target fine-tuning task. While originally formulated in the context of transfer learning for fine-tuning pretrained models to specific tasks, it has also proven effective in protecting against catastrophic forgetting and improving data efficiency in domain adaptation [START_REF] Zhang | Unsupervised domain adaptation with adapter[END_REF].

Elastic Weight Consolidation (EWC) [START_REF] Kirkpatrick | Overcoming catastrophic forgetting in neural networks[END_REF]) is a CF mitigation technique which consists of penalizing updates to parameters proportionally to how critical they are for the original task and data distribution for which the model was optimized. This has been used to some success in Transformers [START_REF] Arumae | CALM: Continuous Adaptive Learning for Language Modeling[END_REF][START_REF] Xu | Forget me not: Reducing catastrophic forgetting for domain adaptation in reading comprehension[END_REF], but is by construction less effective at reducing CF than Adapters for the benefit of better parameter efficiency.

. Knowledge Integration

Knowledge-based systems have been used in conjunction with neural networks since the beginnings of multi-layer neural networks, for general purposes [START_REF] Oliver | Using rules and task division to augment connectionist learning[END_REF] as well as for a variety of specific tasks such as robotic agent control [START_REF] Handelman | Integrating neural networks and knowledge-based systems for intelligent robotic control[END_REF] and gene recognition in DNA sequences [START_REF] Towell | Refinement of approximate domain theories by knowledge-based neural networks[END_REF][START_REF] Noordewier | Training knowledge-based neural networks to recognize genes in dna sequences[END_REF]. This approach has mostly fallen out of favor since the deep learning resurgence of 2012 as progress has continued on neural network architectures and training procedures. Some work on this topic has continued however, and influenced the current landscape of the field of knowledge integration as it undergoes a resurgence of its own.

In contrast to typical domain adaptation techniques, knowledge integration, sometimes referred to as Knowledge-Infused Learning [START_REF] Kursuncu | Knowledge infused learning (k-il): Towards deep incorporation of knowledge in deep learning[END_REF][START_REF] Sheth | Shades of knowledgeinfused learning for enhancing deep learning[END_REF][START_REF] Wang | K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters[END_REF][START_REF] Yuan | CODER: Knowledgeinfused cross-lingual medical term embedding for term normalization[END_REF], relies on making use of available a priori knowledge. The core trait of this type of approach is that the neural model must make use of the prior knowledge, improving the bottom-up learning procedure with a top-down knowledge-based component. These approaches therefore do not include:

• using prior knowledge of a problem to decompose the learning objective into multiple sub-tasks as described by [START_REF] Oliver | Using rules and task division to augment connectionist learning[END_REF] • neuro-symbolic approaches such as proposed by [START_REF] Kroshchanka | A neural-symbolic approach to computer vision[END_REF] which augment relatively general knowledge-based architectures with narrower neural subsystems More generally, we do not consider that using prior knowledge of the problem in order to devise a fully bottom-up model architecture or overall top-down AI system which is more effective or data-efficient than learning the task end-to-end constitutes Knowledge Integration.

In addition to its applicability to the enhancement of general language models at the pre-training stage, this approach has multiple advantages. Depending on the implementation of the integration, the model may be able to take advantage of knowledge base concepts and relations never encountered in training, allowing the model to progress without additional training as KBs are updated and reducing the amount of in-domain concepts that must be covered by the training corpus. As KBs are a much denser source of information than raw text, effectively leveraging them may also potentially reduce the amount of in-domain text required compared to that of extended-pre-training-based methods. In addition, models which require less in-domain training data are not only applicable to a wider variety of domains and less costly to deploy, they are also less sensitive to catastrophic forgetting (Piat et al., 2022a). Lastly, use of structured knowledge can help with explainability [START_REF] Gaur | Semantics of the black-box: Can knowledge graphs help make deep learning systems more interpretable and explainable[END_REF], which is critical in high-stakes applications such as healthcare, finance and education.

Knowledge may take multiple forms such as rules or heuristics, but the most common approach is to use pre-existing KBs. Typically, this is done by leveraging the KB at the pre-training stage, meaning the domain adaptation step needs only be carried out once to benefit all of the various downstream tasks. In general language models, this is usually achieved by identifying knowledge base concept occurrences (or "mentions") in the input text, extracting relevant concept and/or relation embeddings from the knowledge graph which are precomputed using a graph embedding algorithm such as TransE [START_REF] Bordes | Translating embeddings for modeling multi-relational data[END_REF], and using the knowledge embeddings to improve the language model's distributed word representations.

There are three main factors which characterize knowledge integration methods:

• The type of prior knowledge: typically rules, heuristics, or knowledge graphs.

• The method of integration: typically as additional input to the network, as an external resource, or as a mechanism for altering the network architecture or loss-function.

• The representation of the knowledge, which is most commonly vectorial, but often tailored to the model based on the previous two factors.

In this section, we delve into the various approaches that have been attempted and further elaborate on the landmark articles which have influenced the direction of our work and of the field as a whole. First, we will go over non-KG-based approaches and the relevant knowledge representations. Next, as KGs are almost exclusively used in language modeling, we will go over the dominant approaches to KG integration in RNN based LMs. Lastly, we will discuss the approaches used in Transformer-based LMs.

. Overview of common knowledge types and related approaches

In The Integration of A Priori Knowledge into a Go Playing Neural Network, Enzenberger (1996) provide a good example of the use of rules and heuristics in the form of "experts" to augment the capabilities of a Go-playing neural network. An "external expert" implements simple known patterns and strategies which can override the decisions of the network in well-documented board configurations. A "feature expert" alleviates the burden of learning useful patterns by implementing prior known types of patterns and feeding a higher-level representation directly to the neural network. Lastly, a "relation expert" imposes a direct relation between neurons and elements on the board, allowing connections only between neurons corresponding to elements that are adjacent on the board, which change as the game progresses. Trained with self-play, this approach was shown to be more effective than both neural networks and rule-based systems of the time. This type of knowledge integration which informs or alters the very structure of the neural network has been explored in other projects [START_REF] Neal | Bayesian learning for neural networks[END_REF][START_REF] Piat | Refining rules of emotion recognition in hybrid systems[END_REF][START_REF] Towell | Knowledge-based artificial neural networks[END_REF][START_REF] Tan | Cascade artmap: Integrating neural computation and symbolic knowledge processing[END_REF] but this type of approach is more difficult to put in place with the worse explainability of current, larger models. Some domain knowledge may take the form of logic rules. These have also been used to construct neural network topologies such as for the Knowledge-Based Artificial Neural Networks proposed by [START_REF] Towell | Knowledge-based artificial neural networks[END_REF]. In this approach, the key step is the rules-to-network translator, which defines the topology of the neural network as well as the parameters such that the predictions of the neural network are consistent with the rules. Additional parameters are then added, to account for unknown rules, and the resulting neural network is trained as per normal with the exception of some parameters being frozen if they correspond to known hard rules which must not be altered. [START_REF] Fu | Knowledge-based connectionism for revising domain theories[END_REF] puts forth a similar idea, with a reverse network-to-rules translation element which additionally enables extraction of new symbolic rules from the neural network. [START_REF] Tan | Cascade artmap: Integrating neural computation and symbolic knowledge processing[END_REF] proposes a roughly equivalent approach, but in the context of Adaptive Resonance Theory.

Inductive Logic Programming (ILP) [START_REF] Muggleton | Inductive logic programming[END_REF] is a similar machine learning approach which uses prior logical knowledge, typically expressed as Horn clauses, including positive statements, negative statements and constraints, in order to produce a logic program which results in all positive statements being true and negative statements being false, with constraints typically implemented as part of the network architecture (d'Avila [START_REF] Garcez | The connectionist inductive learning and logic programming system[END_REF]. ILP is often tackled using neural networks, which require a vector as input. The vectorization scheme for clauses is typically propositionalization, which is thematically similar to embedding [START_REF] Lavrač | Propositionalization and embeddings: two sides of the same coin[END_REF] but uses a boolean value range. While initially based on manually engineered features, automated feature construction for propositionalization was introduced by [START_REF] Lavrač | Learning nonrecursive definitions of relations with LINUS[END_REF].

Domain Specific Languages (DSLs) are computer languages which are designed for a specific goal, such as HTML for describing web pages, or AWK for text processing. Some DSLs are constructed to be used specifically by a programinduction algorithm (and as such constitute a form of prior knowledge), some of which are based on neural networks such as the work of [START_REF] Lake | Human-level concept learning through probabilistic program induction[END_REF] on handwritten digit recognition and generation, and of [START_REF] Devlin | Neural program meta-induction[END_REF] on robotic motion. DREAMCODER [START_REF] Ellis | Dreamcoder: Bootstrapping domain-specific languages for neurally-guided bayesian program learning[END_REF] additionally learns to construct the appropriate DSL.

Some knowledge integration makes use of their prior knowledge in the loss function. The simplest way to achieve this is generally to introduce a loss penalty informed by prior knowledge [START_REF] Fischer | DL2: training and querying neural networks with logic[END_REF][START_REF] Muralidhar | Incorporating prior domain knowledge into deep neural networks[END_REF]. On an abstract level, one would use prior knowledge to formulate constraints and penalize model predictions which violate these constraints. This is often done using knowledge graphs. For instance, [START_REF] Rocktäschel | Low-dimensional embeddings of logic[END_REF] extract hypernymy information from WordNet and induce the modeling of these relations by penalizing scenarios where statements which are true are predicted to be false when the entities mentioned are replaced by hypernyms. [START_REF] Takeishi | Knowledge-based distant regularization in learning probabilistic models[END_REF] provide a loss function regularization for continuous data which assumes a knowledge graph exists which covers the various features as concepts and that related features have similar distributions in the dataset.

Other approaches use logic rules, such as work by [START_REF] Xu | A semantic loss function for deep learning with symbolic knowledge[END_REF], for instance, who devise a penalty term which models the likelihood that a given model output satisfies a set of logical constraints. In a similar setup to Iterated Distillation and Amplification [START_REF] Christiano | Supervising strong learners by amplifying weak experts[END_REF][START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], whereby a model is trained to imitate an amplified version of itself, [START_REF] Hu | Harnessing deep neural networks with logic rules[END_REF] propose to train a model to imitate a rule-constrained version of itself in order to distill rules into the neural architecture.

Prominent examples of the approach by loss function alteration in Transformerbased language models include UmlsBERT [START_REF] Michalopoulos | UmlsBERT: Clinical domain knowledge augmentation of contextual embeddings using the Unified Medical Language System Metathesaurus[END_REF], which we will go over in Section 2.5.4, and KEPLER (Wang et al., 2021b) which optimizes both a masked language modeling objective and a knowledge embedding objective which uses text descriptions of concepts to construct concept embeddings and enforces desirable geometric properties to derive relation embeddings similarly to TransE [START_REF] Bordes | Translating embeddings for modeling multi-relational data[END_REF] (see Section 2.5.2).

. KG-based approaches

Knowledge Graphs exist for many domains and generally contain widely applicable prior knowledge. They are also generally reliable, fairly standardized and non-proprietary. This makes them good candidates for knowledge integration, particularly for domain-specific tasks, and are thus used in a variety of methods and applications in the literature.

Perhaps the most obvious lead in terms of processing Knowledge Graphs would be using Graph Neural Networks (GNNs), a family of neural networks whose architectures are designed to process data organized as graphs. This approach has, for instance, been used to pre-train a discriminator in a Generative Adversarial Network (GAN) setup for drug design [START_REF] Dash | Using domain-knowledge to assist lead discovery in early-stage drug design[END_REF]; as a knowledge-based component of a multi-expert system predicting the answer to a Visual Question-Answering (VQA) problem [START_REF] Marino | Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa[END_REF]; or to fuse text and KG-based inputs in Transformers [START_REF] Zhang | GreaseLM: Graph REASoning Enhanced Language Models for Question Answering[END_REF][START_REF] Yasunaga | Deep bidirectional language-knowledge graph pretraining[END_REF]. GNNs are most popular in graph-oriented tasks however, as they lack flexibility in terms of their ability to interface with other modalities. In the context of knowledge graphs, they are therefore more suited to tasks such as knowledge graph completion, concept classification, or knowledge graph merging, making them a generally less compelling processing option than graph embeddings when supporting other tasks for which the knowledge graph is not the central component. For further review of the literature, we direct readers to the work of [START_REF] Ye | A comprehensive survey of graph neural networks for knowledge graphs[END_REF] for the use of GNNs with Knowledge Graphs, and of [START_REF] Wu | Graph neural networks for natural language processing: A survey[END_REF] for a more comprehensive overview of the use of GNNs in NLP.

In Leveraging Lexical Resources for Learning Entity Embeddings in Multi-Relational Data, [START_REF] Long | Leveraging lexical resources for learning entity embeddings in multi-relational data[END_REF] use pretrained Word2Vec embeddings [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] and Knowledge Base concept descriptions as an initialization for the TransE knowledge graph embedding algorithm [START_REF] Bordes | Translating embeddings for modeling multi-relational data[END_REF]. As such, they are effectively modifying word embeddings so as to take information from the knowledge base into account. TransE achieves this by imposing that, for a triple (c 1 , r, c 2 ) where c 1 and c 2 are concept nodes and r a relation edge, their corresponding vectors are bound by the quasi-equality c 1 + r ≈ c 2 . This is consistent with existing known properties of the Word2Vec embedding space, e.g.

--→ king ---→ man+ -----→ woman ≈ ---→ queen or ----→ f rance+ ----→ capital ≈ ---→
paris where ---→ word denotes the embedding for "word". The concept and relation embeddings can therefore be used in place of their pretrained word embedding counterparts in addition to func-tioning as node and vertex embeddings. [START_REF] Celikyilmaz | Enriching word embeddings using knowledge graph for semantic tagging in conversational dialog systems[END_REF] follow a similar method to achieve the same objective, but find that adding a concurrent language modeling objective and adding neighboring concepts to the context improves performance. No mechanistic explanation for this improvement is given, but it is likely that the language modeling objective reduces catastrophic forgetting. [START_REF] Wang | Incorporating linguistic knowledge for learning distributed word representations[END_REF] attempt to achieve a similar result by integrating KG relation information in the word embedding objective. These approaches are limited however, as the cases for multi-word concepts, synonyms of concepts unaccounted for in the KB, homographs, etc. are non-trivial to account for and can be quite common depending on the knowledge graph. Furthermore, with fixed-vocabulary models, not all concepts in the knowledge graph may have a corresponding pretrained embedding, limiting the usefulness of the method in a domain adaptation setting. While this approach, unlike many others, does not require Knowledge Base concepts to be identified in the text, the reverse problem applies: one must still automate the matching of KB concepts and relations with existing embeddings, which would likely introduce many mistakes of the same variety. Lastly and most significantly, these methods rely on embeddings being static and are therefore not directly applicable to contextualized word embeddings.

For further in-depth discussion on the aforementioned approaches, readers are directed toward the work of [START_REF] Dash | A review of some techniques for inclusion of domain-knowledge into deep neural networks[END_REF]

. In RNN LMs

NKLM

In A Neural Knowledge Language Model, [START_REF] Ahn | A neural knowledge language model[END_REF] propose a mechanism which enables a generative LSTM-based LM to choose whether to generate a word from its vocabulary or from a Knowledge Graph. The KG is represented as a set of "facts" F, i.e. triples a = (c a,1 , r a , c a,2 ) where nodes c a,1 and c a,2 are concepts (respectively the "subject" and "object" of the fact) and edge r is a relation. Embeddings for concepts and relations are derived using TransE [START_REF] Bordes | Translating embeddings for modeling multi-relational data[END_REF] and fixed in order to allow the model to generalize to new facts. They are packed into a fact matrix F where each column a i = c a i ,1 ⊕ r a 1 ⊕ c a i ,2 where ⊕ is the vector concatenation operator.

As shown in Figure 2.5, the Neural Knowledge Language Model (NKLM) uses a regular LSTM architecture and uses the state vector h t at time-step t to determine which fact to select from F using an MLP implementing the f factkey (h t , e) function where e is a mean-pooling of F. This returns a fact index a t which can be used to retrieve the most relevant fact embedding a t . This fact embedding is then used, in conjunction with h t , to decide whether the next word should be generated from the vocabulary or the knowledge base using f copy (h t , a t ), similarly implemented as an MLP. In either case, h t and a t are fed to an MLP implementing an index look-up similarly to f factkey : f voca and f poskey for the vocabulary-and knowledge-based generation respectively.

f voca straightforwardly yields a key to look-up the next word in the word embedding matrix (alternatively, this can be seen as predicting the embedding for the next word and retrieving the closest true word embedding), and f poskey has the equivalent function in knowledge-based word indexing, using trained position embeddings which refer to words by their position in the description of c at,2 , the object of fact a t . As knowledge-word position embeddings do not share the same vector space as regular word embeddings, both a word embedding and position embedding, in addition to the previously generated fact embedding a t , are fed to the LSTM at every time step t, with the embedding that was not generated at step t -1 initialized to the ⃗ 0 vector.In their paper, Ahn et al. add a layer of probabilistic interpretation to the retrieval mechanism which yields a t , w o t and w v t by applying a Sof tmax before argmax. We omit this from figure 2.5 for simplicity as Sof tmax has no effect on the output of argmax.

P[w o ] t w o ; t w v ← t t t t w v W[w v ] t a t-1 w v w o t-1 t-1 k pos W W or d em b ed di ng m at ri x • • k voc NKLM • F Fa ct em b ed di ng m at ri x
This approach, by using position embeddings, treats the KB as an external resource which can be queried. It has the advantage of allowing the model to generalize to new facts and words added to the KB after training and, in a traditional word-based vocabulary, to predict out-of-vocabulary words. It suffers from many shortcomings, however. In particular, each word can only be associated with one fact. Ahn et al. present a limited situation in which the text is focused on one known topic τ which is the subject (c a,1 ∀a ∈ F τ with F τ the set of facts related to topic τ ) and therefore cannot have multiple facts per word unless the knowledge graph has pairs of concepts that share more than one relation (that is, pairs of vertices that share more than one edge in a given orientation). In deployment, the ability to change topic focus mid-text or perhaps focus on multiple topics simultaneously is highly desirable. Furthermore, the indexing approach for knowledge-based token generation likely works because the KGs used have relatively few relation types and concept descriptions tend to be short and homogeneous. This approach is therefore unlikely to scale to larger, more complex KGs well. Lastly, this method heavily relies on string matching in order to construct supervised training samples and prompts. Depending on the knowledge base and topic, this may be unsatisfactory as, in addition to the potentially high computational cost of the approach, homographs may match incorrect concepts and synonyms or paraphrases may fail to match mentioned concepts. This problem is known as Entity Linking and current methods continue to struggle in all but the most trivial of cases (see section 2.6).

KBLSTM

In Leveraging Knowledge Bases in LSTMs for Improving Machine Reading, [START_REF] Yang | Leveraging knowledge bases in LSTMs for improving machine reading[END_REF] propose the Knowledge-enhanced Bidirectional LSTM (KBLSTM), a concept-embedding-based method which relaxes the problem of Entity Linking by retrieving multiple candidate concepts per word from the KG. Specifically, candidate concepts which may be related to the current word are retrieved, and their embeddings are pooled with the output of the Bi-LSTM at each time step. The relevance to the current context is estimated using attention, in addition to a "sentinel" vector which allows the model to focus on the context and ignore the contextless background knowledge where relevant by deciding how much weight to assign to KG information versus the current state.

The KG is represented as a set of triples (c 1 , r, c 2 ) with c 1 , c 2 being any two concepts (nodes) with relationship (edge) r. Knowledge Graph Embeddings v c for each concept c are derived following [START_REF] Yang | Embedding entities and relations for learning and inference in knowledge bases[END_REF].

As depicted in figure 2.6, the KBLSTM replaces the hidden state vector h t with:

ĥt = h t + m t
where m t is a "knowledge state vector" which contextualizes knowledge vectors with respect to h t . It is computed as a mixture model, allowing for a better tradeoff between the impact of background knowledge and information from the context: Where V (x t ) is the set of candidate concepts from the KB matching x t , α ti acts as an attention weight which reflects the relevance of the concept i, and β t is the weighting term for the local context. Together, α ti and β t form a probability distribution and are defined as:

m t = i∈V (xt) α ti v i + β t s t
α ti ∝ exp(v T i W v h t ) β t ∝ exp(s T t W s h t ) i∈V (xt) α ti + β t = 1
s t is the aforementioned sentinel vector which encodes the context information and determines the knowledge/context tradeoff, defined as2 :

b t = σ(W b h t-1 + U b x t ) s t = b t ⊙ tanh(c t )
The word embeddings x t are pretrained following [START_REF] Wieting | Towards universal paraphrastic sentence embeddings[END_REF] andfine This article marks the first significant usage of a KG to improve a language model at the embedding level. This approach elegantly matches entity mentions to KB concepts and learns to combine the information from both modalities. It does, however, have several weaknesses. First, as m t is not explicitly part of the h t space, the responsibility for alignment of vector spaces falls on the optimization of the entire model for each task. It is generally preferable to learn projections between vector spaces explicitly, as it would make this method more easily applicable to fine-tuning pretrained language models and incentivize parameter specialization in the model. Furthermore, the candidate concept generation process does not work well for multi-word concepts ; while the approach to Entity Linking described has the advantage of being fuzzy thus allowing the model to learn which candidates are likely to be relevant, the single-word search is likely to produce low-precision results making learning difficult, and the lack of context in the search is likely to cause relatively low recall on some KBs. Despite these shortcomings, the approach has been influential, inspiring approaches such as KnowBert [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF], KAGNET [START_REF] Lin | Kagnet: Knowledge-aware graph networks for commonsense reasoning[END_REF] and SenticLSTM [START_REF] Ma | Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM[END_REF].

. In Transformer LMs

In the interest of reducing the pre-training burden of Transformer-based LMs such as BERT and expanding the range of concepts and relationships that they can accurately predict without resorting to additional freeform text, multiple methods for incorporating knowledge in contextualized word embeddings have been developed. One of the main categories of approaches is to rely on the Transformer's attention mechanism to combine entity and word information, as do ERNIE [START_REF] Zhang | ERNIE: Enhanced language representation with informative entities[END_REF], K-Adapters [START_REF] Wang | K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters[END_REF] and KnowBert [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF]. K-Adapters in particular are compelling as they use Adapters (see Section 2.4.5) which, by construction, prevent catastrophic forgetting. Another common type of approach is to train a neural network or the language model itself to align entity representations with token representations, whether they be the input WordPiece representations such as E-BERT [START_REF] Poerner | Efficient-Yet-Effective Entity Embeddings for BERT[END_REF] or contextualized output representations such as KEPLER (Wang et al., 2021b) and CODER [START_REF] Yuan | CODER: Knowledgeinfused cross-lingual medical term embedding for term normalization[END_REF]. UmlsBERT [START_REF] Michalopoulos | UmlsBERT: Clinical domain knowledge augmentation of contextual embeddings using the Unified Medical Language System Metathesaurus[END_REF], in contrast, does not fit into the aforementioned categories as it mainly consists of biasing the BERT input vectors for entity mentions with a topic vector, and changing the Cross-Entropy loss in the Masked Language Modeling (MLM) objective to a Binary Cross Entropy Loss, setting all synonyms of a medical term as valid targets.

More specifically, E-BERT learns a projection of entity embeddings derived from the KB to the input word embedding space which preserves KG relation information, enabling their use in input text and providing the model with more information. UmlsBERT, on the other hand, explicitly adds semantic group embeddings to words found in UMLS. ERNIE and BERT-MK [START_REF] He | Integrating graph contextualized knowledge into pre-trained language models[END_REF] learn a fused representation of contextualized word and entity representations. K-Adapters [START_REF] Wang | K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters[END_REF] are a computationally cheaper alternative to the ERNIE process which avoid the problem of catastrophic forgetting by using Adapters [START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF], i.e. adding layers within the transformer which can be enabled or disabled and take on the burden of fine-tuning while the rest of the model weights remain frozen. While these methods have largely been successful, one drawback which we discuss in section 2.6 is that they all require a separate upstream Entity Linking (EL) step to be made at inference, limiting their performance and scope of applicability. Some approaches do not suffer from the need for an EL step. KEPLER (Wang et al., 2021b) is one such model, which introduces a knowledge embedding loss as an objective for language model pre-training, aligning contextual word representations with entity description representations. As such, it does not require an EL step at inference. The KnowBert model developed by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF], on the other hand, grafts a KB-specific EL module into a transformer-based pretrained LM such as BERT, in order to jointly perform EL and contextualized word representation enrichment, making it also a standalone method requiring no upstream EL with the additional benefit of explicitly identifying the entities present in the text.

This distinction between methods which do or do not require an upstream EL step to be performed, that is, whether they require entities mentioned in the input text to be precisely identified, is an important one. This property is in fact one of the most important criteria for selecting a knowledge integration method in the biomedical case, and with UMLS in particular, as EL is currently an unsolved problem. On the MedMentions corpus [START_REF] Mohan | MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts[END_REF], for instance, the best performing models are MedCat [START_REF] Kraljevic | Multi-domain clinical natural language processing with medcat: the medical concept annotation toolkit[END_REF] and Bhowmik et al.'s dual encoder [START_REF] Bhowmik | Fast and effective biomedical entity linking using a dual encoder[END_REF] which achieve 0.448 and 0.534 F 1 score respectively. The Entity Linker used by UmlsBERT is cTAKES [START_REF] Savova | Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications[END_REF], which reaches an F 1 score of 0.178 on MedMentions, as reported by Kraljevic et al.. The majority of the aforementioned knowledge integration approaches are subject to this limitation. Whilst KEPLER and CODER are exceptions which do not require an upstream EL step at inference, CODER does require this in training, and KEPLER is trained on descriptions of each entity in the knowledge base, which are not available for some KBs such as UMLS. KnowBert relaxes the EL requirement, calling only for candidate entity mentions, and is trained both on Language Modeling and Entity Linking. KnowBert is thus less limited by the EL performance than other knowledge-based models, and does not require as much in-domain text as pre-training-based approaches. It therefore has considerable potential to effectively utilize the KB and is unlikely to suffer from CF on general text.

Several models are worth reviewing in greater detail due to their influence on the field as well as my work.

ERNIE [START_REF] Zhang | ERNIE: Enhanced language representation with informative entities[END_REF] (not to be confused with the identically named model by [START_REF] Sun | ERNIE: Enhanced representation through knowledge integration[END_REF] which uses knowledge to simply preferentially mask entities in the MLM objective) is one of the first, simplest and most influential approaches, having inspired approaches such as BERT-MK [START_REF] He | Integrating graph contextualized knowledge into pre-trained language models[END_REF], GREASELM [START_REF] Zhang | GreaseLM: Graph REASoning Enhanced Language Models for Question Answering[END_REF] and KnowBert [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF]. It is based on the BERT model, but substitutes some of the basic transformer encoder blocks with knowledge-enhanced encoders called "K-Encoders" (see Fig. 2.7). K-Encoders accept additional inputs and outputs in the form of entity embeddings corresponding to entities mentioned in the sequence. The entity embeddings are fed through a multi-head attention mechanism which is identical to but separate from the multihead attention applied to tokens. These recontextualized entity embeddings ẽk are then paired with their corresponding recontextualized token embeddings wj and fed to a token-entity information fusion mechanism which projects the embeddings into a unified space, performs their sum, and projects the fused representation back to token and entity space resulting in text-enriched entity representations e k and knowledge enriched token representations w j (see Eq. 2.8). GREASELM [START_REF] Zhang | GreaseLM: Graph REASoning Enhanced Language Models for Question Answering[END_REF] is a similar approach which, rather than relying on pre-existing entity embeddings, extracts a relevant subgraph of the KG given the input text, and substitutes an attention-enabled GNN [START_REF] Veličković | Graph attention networks[END_REF] for ERNIE's entity recontextualization step. Subsequently, rather than performing information fusion on matched token and entity embeddings, a sequence embedding similar to the [CLS] token in BERT3 and an equivalent "node interaction embedding" for the graph are used to perform the information fusion indirectly. This side-steps the difficulty of matching multi-word entity names to multiple input tokens, promotes information mobility, and is computationally more efficient at inference. Both ERNIE and GREASELM suffer from the requirement of an upstream Entity Linking step, however GREASELM's constraint is weaker than ERNIE's due to the lack of necessity to match token and entity embeddings.

h j = σ( W t wj + W e ẽk + b) (2.8) wj = σ(W t h j + b t ) (2.
DRAGON [START_REF] Yasunaga | Deep bidirectional language-knowledge graph pretraining[END_REF] significantly improves on GREASELM by introducing a graph completion self-supervised pre-training objective which masks relations in the knowledge sub-graph, similarly to the standard MLM objective. UmlsBERT [START_REF] Michalopoulos | UmlsBERT: Clinical domain knowledge augmentation of contextual embeddings using the Unified Medical Language System Metathesaurus[END_REF] is, as its name suggests, a BERT-based language model which integrates knowledge from UMLS. It uses two knowledge integration mechanisms in tandem: knowledge augmentation of input embeddings, and a knowledge-informed multi-label MLM objective. The first mechanism consists of classifying each clinical input token into one of UMLS' "semantic types" 4 (groups of semantically related concepts) and adding to the token, segment and position embeddings 3 the relevant semantic type embedding. This effectively stretches input embeddings of the same semantic type in the same direction in the input space, giving BERT information on the relatedness of the tokens, which is particularly useful with uncommon words. The second mechanism uses UMLS' thesaurus feature, which associates clinical concepts to a list of names. Rather than using BERT's cross-entropy loss in its MLM objective to generate a probability distribution on the vocabulary, UmlsBERT uses binary cross-entropy -which gives an independent probability for each token to be an acceptable completion -and sets every clinical synonym as defined by UMLS to be an acceptable completion. For instance, for the masked word "lung", "pulmonary" would also be a positive label. This leads the model to learn to associate clinically related words, but should degrade the model's grammatical accuracy. It is not made explicit how the semantic type embeddings are computed, nor how the multi-label objective interacts with WordPiece tokenization and multi-word entity mentions. This model obviously suffers from reliance on accurate Entity Linking, but also uses Clinical-Bert as a pretrained backbone. Having been trained on approximately six times more biomedical text than BERT, accurately quantifying the added value of its knowledge integration is not straightforward.

. Entity Linking

Entity Linking (EL) is a task which consists of identifying spans of input text which refer to specific entities or concepts, and pairing them with the appropriate target they refer to in a knowledge base. Take, for example, the following sentence:

The council of the EU intends to force the operators of secure messaging apps such as WhatsApp or Signal to allow intelligence services to gain access to encrypted conversations through backdoors.

Assume we use Wikipedia as our KB. We thus wish to identify any entity which has a Wikipedia page, and a unique identifier from which we can look up the page in question (for our purposes, we will use the page URL) for each mentioned entity. The entities in question for the aforementioned example are the council of the EU (1), the European Union (2), WhatsApp (3), Signal (4), Intelligence Agencies (5), and Backdoor (6). As a result of the Entity Linking process, we would like the occurrences of the aforementioned entities to be identified and labeled. We illustrate this in the following sentence by enclosing entity mentions in boxes and adding corresponding numbers (1-6) as subscripts.

The council of the EU 2 1 intends to force the operators of secure messaging apps such as WhatsApp 3 or Signal 4 to allow intelligence services 5 to gain access to encrypted conversations through backdoors 6 .

Each of the labels (1-6) should be matched to a URL as per Table 2 In summary, in our definition5 , Entity Linking consists of two separate tasks: Mention Detection (MD; a KB-specific NER task which does not require an entity type) and Entity Disambiguation (ED; also referred to as Entity Normalization). The additional task of Candidate Generation (CG), i.e. a pre-selection of potentially relevant concepts, is often inserted between MD and ED. Crucially, an Entity Linking task is accompanied by a corresponding KB which defines which entities must be marked. For all practical purposes, regardless of the KB, we may assume that each entity is associated with at least one plain text descriptive "name". We may not, however, assume that entities systematically have a textual description. Because EL is dependent on a specific KB, it is difficult to systematically compare performance across methods. We will therefore go over the main approaches in general terms, before focusing on specific contributions for the biomedical domain, contextualizing the work described in subsequent chapters of this thesis.

The naïve approach to EL consists of matching every substring of the input text to every name for every entity in the KB using a string similarity measure such as Levenshtein distance or Jaccard similarity and selecting a threshold above which the span is linked to the relevant entity. This is the approach followed, for instance, by QuickUMLS [START_REF] Soldaini | Quickumls: a fast, unsupervised approach for medical concept extraction[END_REF]. RysannMD [START_REF] Cuzzola | RysannMD: A biomedical semantic annotator balancing speed and accuracy[END_REF] and [START_REF] Rao | Entity linking: Finding extracted entities in a knowledge base[END_REF] use string matching as a basis for candidate selection before applying more sophisticated final selection and disambiguation procedures. Several approaches [START_REF] Han | Collective entity linking in web text: a graph-based method[END_REF]Hoffart et al., 2011a;[START_REF] Alhelbawy | Collective named entity disambiguation using graph ranking and clique partitioning approaches[END_REF] propose to generate candidate matches in this way, extract the subgraph of mentioned entities from the KG, and differ in their method for selecting only the most highly related nodes. While many of these approaches show fairly compelling results, this exhaustive approach works only for small amounts of text and small KBs, and quickly becomes intractable as we discuss in Chapter 4. Additionally, the use of string similarity as a basis for selecting candidate mentions yields poor results for many KBs which do not provide sufficiently many different names for their entities. Using our previous example, it is unlikely that an occurrence of the word "EU" could be accurately matched to the page of the "European Union" using string similarity. [START_REF] Kolitsas | End-to-end neural entity linking[END_REF] use an ELMo-like BiLSTM language model to represent their input text as contextualized word embeddings, which they pool into span embeddings for all sub-sequences of the input text exhaustively. They then retrieve a number of candidate entity embeddings from a pretrained probabilistic map provided by [START_REF] Ganea | Deep joint entity disambiguation with local neural attention[END_REF] for each span embedding. They then measure the similarity between the span embeddings projected in the entityembedding space and the candidate embeddings and use this similarity score as a basis for their EL decision. This approach is essentially a vectorial version of the aforementioned approaches, and as such benefits from better hardware acceleration, but shares the same fundamental computational complexity. Furthermore, the pretrained span-to-entity-embedding map must be created for every knowledge graph, which poses its own concerns in terms of tractability and requirement for annotated text.

Van Hulst et al. (2020) use ELMo-based contextualized word embeddings to power the Mention Detection step, modeled as a simple NER task. Candidate entities are chosen in the same way as [START_REF] Kolitsas | End-to-end neural entity linking[END_REF] and [START_REF] Ganea | Deep joint entity disambiguation with local neural attention[END_REF] and/or by cosine similarity between entity embeddings and the sum of the words in the context of the detected mention ("contextual similarity") (which assumes entity and word embeddings are in the same space). Entities are disambiguated according to a score computed as a function of prior probability, contextual similarity, and "coherence" [START_REF] Ganea | Deep joint entity disambiguation with local neural attention[END_REF][START_REF] Le | Improving entity linking by modeling latent relations between mentions[END_REF]) (a measure of relatedness between different entity candidates for the span). The lower complexity of the MD step and the fewer candidates selected compared to previous methods improve the complexity of this approach, but it is specific to the Wikipedia KB and heavily relies on the work of [START_REF] Ganea | Deep joint entity disambiguation with local neural attention[END_REF], making it difficult to adapt to other KBs.

The obvious way to avoid the need for a span-to-entity-embedding map such as the one created by [START_REF] Ganea | Deep joint entity disambiguation with local neural attention[END_REF] is to embed entities and tokens in the same space. [START_REF] Moreno | Combining word and entity embeddings for entity linking[END_REF], for instance, propose an extension of Word2Vec which applies to corpora containing entity mentions, as well as several similarity measures to perform ED. These approaches rely on the existence of such annotated corpora however, which are not available for most knowledge bases. [START_REF] Ravi | CHOLAN: A modular approach for neural entity linking on wikipedia and wikidata[END_REF] uses two Transformer-based LMs in sequence. The first LM performs MD as a typical NER task, feeding the resulting spans to a wikipediaspecific search engine which incorporates a lookup table for related terms and a page ranking algorithm. The search engine returns candidate entities, and the second LM performs ED by modeling it as a sequence classification task: it is fed the detected mention, the context sequence, and a description of the candidate entity, and must predict whether the candidate entity matches the mention. While the approach is efficient and performs well on Wikipedia, the reliance on its search engine and assumption that entities are accompanied by descriptions makes it inapplicable to most other KBs. Additionally, it requires an annotated corpus to train the ED task, which is rarely available. E-BERT [START_REF] Poerner | Efficient-Yet-Effective Entity Embeddings for BERT[END_REF], which is originally a Knowledge-Integration approach and hypothetically requires an upstream EL step, proposes an ED approach which leverages its concept-embedding-to-word-embedding projection such that it can perform its own EL (assuming the MD and CG steps are already handled). While E-BERT normally expects concept embeddings to be projected to the WordPiece input space and fed as input simultaneously with regular input Word-Piece embeddings (all the while assuming that the relevant knowledge is not lost in the projection process), Poerner et al. propose to repurpose the mechanism in reverse. Specifically, they suggest to find concepts by replacing their embeddings in the input with an average of the candidate concepts' embeddings, and training the model to output the embedding for the correct concept. While at first glance this approach may seem like it relies on the language model's knowledge thus defeating the purpose of knowledge integration, the upstream CG process and the gap between the true and predicted concept embedding contribute knowledge that the model does not necessarily possess. This approach unfortunately relies on MD and CG tools as well as large amounts of text with annotated KB concepts, reducing its scope of applicability to the Wikipedia KB, and few if any others.

End-to-end neural Entity Linking had historically not been heavily considered until the introduction of the Transformer. This approach has since been attempted by [START_REF] Broscheit | Investigating entity knowledge in BERT with simple neural end-to-end entity linking[END_REF] and [START_REF] Chen | Contextualized End-to-End Neural Entity Linking[END_REF], which we will go over in Chapter 3. Transformers have also enabled approaches for matching entity mentions to entity embeddings in a shared vector space [START_REF] Humeau | Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multisentence scoring[END_REF][START_REF] Logeswaran | Zeroshot entity linking by reading entity descriptions[END_REF][START_REF] Wu | Scalable zero-shot entity linking with dense entity retrieval[END_REF], assuming that textual descriptions of mentions exist. This assumption is not verified for all KBs, and notably not for UMLS; furthermore, these approaches tend to be computationally expensive due to the necessity of mining hard negative samples, though [START_REF] Wiatrak | Proxy-based zero-shot entity linking by effective candidate retrieval[END_REF] have introduced a loss regularization which helps relax this constraint.

While Entity Linkers are known not to be perfectly reliable, Biomedical Entity Linking on the UMLS KB is a particularly difficult task. One of the most widely used biomedical EL systems is cTAKES [START_REF] Savova | Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications[END_REF], which reaches an F 1 score of 0.178 on MedMentions, as reported by Kraljevic et al. Some of the best performing models include RysannMD [START_REF] Cuzzola | RysannMD: A biomedical semantic annotator balancing speed and accuracy[END_REF], which achieves 0.436 F 1 on the CRAFT corpus, and the dual encoder architecture proposed by [START_REF] Bhowmik | Fast and effective biomedical entity linking using a dual encoder[END_REF] which achieves 0.534 F 1 score (with 0.529 Precision and 0.538 Recall) on the MedMentions corpus [START_REF] Mohan | MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts[END_REF]. In the latter case, this translates to slightly more than half of the actual entities mentioned being correctly identified, and nearly half of the entities found by the linker being false positives.

49

3 -Biomedical Entity-Linking with Transformers

. Introduction

Whilst knowledge integration has been fairly well explored with projects such as NKLM [START_REF] Ahn | A neural knowledge language model[END_REF], ERNIE [START_REF] Zhang | ERNIE: Enhanced language representation with informative entities[END_REF], or UmlsBERT [START_REF] Michalopoulos | UmlsBERT: Clinical domain knowledge augmentation of contextual embeddings using the Unified Medical Language System Metathesaurus[END_REF], most of the proposed methods assume knowledge-base entities mentioned in the text are known. Requiring human annotation of these concepts defeats the purpose of automation, meaning entity mentions must be automatically identified and labeled upstream of knowledge integration. This task is known as Entity Linking (EL) and, depending on the concepts present in the knowledge base, is typically a difficult task (see Section 2.6).

An obvious approach to EL is to attempt to use a Transformer-encoder language model such as BERT to perform EL by modeling it as a fine-grained Named Entity Recognition (NER) task. That is, given a KB with N concepts, each token in a sequence is classified (based on its output contextualized representation) as one of the N concepts -or as not a mention of any concept. To handle multitoken concept names, the IOB2 scheme is typically applied, with each token being classified as Inside (I) a concept mention, Outside (O) of any concept mention (not-a-KB-concept), or the Beginning (B) of a concept mention1 . Accounting for the B and I tags which can apply to any of the N concepts, the resulting classification has 2N + 1 classes. See Table 3.1 for an example of IOB2 EL tagging of a sequence. This approach has the advantage of being easier to put in place than those mentioned in Section 2.6 and, as it does not contain any string-search or other major CPU-bound operations, can be relatively fast at inference with hardware acceleration, but is limited by a requirement for labeled training data on all concepts and is expensive to train.

INPUT

Pseudomonas aeruginosa Another approach, as the classification process produces a sequence and concept mentions follow a Zipfian distribution much like a vocabulary, is to treat the problem as a sequence-to-sequence task. Given the variety of concepts, each label can be considered an entry in the model's output vocabulary, and the task can be tackled similarly to a translation task for which the target output sequence is the IOB2 "translation" of the input sequence. This is the approach followed by [START_REF] Boguslav | Concept recognition as a machine translation problem[END_REF].

(Pa) infection in OUTPUT B-C0854135 I-C0854135 I-C0854135 I-C0854135 O INPUT cystic fibrosis (CF) patients is OUTPUT B-C0010674 I-C0010674 B-C0010674 O O
To evaluate subsequence classification, we use SeqEval [START_REF] Nakayama | seqeval: A python framework for sequence labeling evaluation[END_REF], which assigns true positives only to perfectly predicted subsequences. Any predicted sequence which does not exactly match a reference sequence from the dataset is considered a false positive, and any reference sequence which does not have a matching prediction is considered a false negative. See Table 3.2 for an example. As SeqEval criteria are stringent, numerically low performance can conceal a large number of near-correct predictions, therefore we also use token-level F 1 (which does not take sequence into account) as an evaluation metric.

Sentence

in cystic fibrosis (CF) patients is Expected: 2 TP O B-C0010674 I-C0010674 B-C0010674 O O FP & 2 FN O B-C0010674 I-C0010674 I-C0010674 O O TP & FN O B-C0010674 I-C0010674 O O O TP & FP & FN B-C0010674 I-C0010674 I-C0010674 B-C0010674 O O

. Training from scratch in a resource-constrained setting

While we originally considered fine-tuning a pretrained model such as BERT, lack of computational power required for fine-tuning BERT-sized language models led us to attempt to train a smaller model from scratch. Based on our computational budget, we set the size of our models to vary circa 8 million parameters (approximately on fourteenth that of BERT BASE ). Expecting poor performance on the MLM task with such a small language model, and expecting full semantic NLU to be unnecessary (as, in order to compete with our baseline string-matching-based approaches such as QuickUMLS [START_REF] Soldaini | Quickumls: a fast, unsupervised approach for medical concept extraction[END_REF], we were expecting token-based fuzzy pattern-matching and some basic intuitions about parts of speech to be sufficient), we initially did not pretrain our models (either on MLM or autoregressively). We attempted the two main aforementioned approaches: the token embedding classification approach, and a sequence-to-sequence approach based on Vaswani et al.'s translation model.

Faced with initial difficulties with models performing marginally better than chance, we created a training curriculum: models were to first be trained on Men-tion Detection (MD) (i.e, the IOB sequence classification task with no entity differentiation required), then on typed NER using UMLS' 127 semantic types (to which each of the concepts in UMLS belong) as entity types, and finally on the Entity Linking task. Word-based, wordpiece-based and character-based tokenization were all attempted, with wordpiece-based tokenization outperforming the alternatives.

Despite Boguslav et al.'s subsequent success with the translation-based approach, likely due to their larger pretrained model, experiments with the sequenceto-sequence model were halted early due to it being more computationally expensive and performing worse than its non-generative counterpart. As shown in Table 3.3, our best performing model performed only marginally better than chance at the MD task. With poorer performance on this simpler task than state of the art entity linkers on the EL task, it became clear that the architecture was not large enough and/or the training data not plentiful enough to learn EL in this manner.

Model

Mention-F 1 Token-F 1 Random predictor 0.00 0.33 Ours 0.01 0.47

. Entity Linking with BioBERT

After having acquired additional computing resources, we attempted to finetune an existing pretrained biomedical language model, BioBERT, to each of the tasks in the aforementioned curriculum (independently, as following the curriculum led to worse results, presumably due to catastrophically forgetting language modeling when learning MD). While MD performance is difficult to evaluate as there exists no baseline for the MedMentions corpus, BioBERT's MD performance exceeded the EL performance of the state of the art model [START_REF] Bhowmik | Fast and effective biomedical entity linking using a dual encoder[END_REF], as depicted in Table 3.4; failure cases mostly involved abbreviations (e.g. "(Pa)" and "(CF)" in the example sequence in Table 3.1).

The Semantic Type NER task appears not to be significantly more challenging than MD for BioBERT. However, increasing the number of classes from 255 (owing to the 127 semantic types tagged in IOB2 scheme) to over 16,000 (with approximately 8,000 concepts in the corpus) revealed that BioBERT's ability to discriminate between UMLS concepts was low. Approximately 94.6% of unique concepts which appear in the MedMentions test corpus were not found as they appeared too rarely in the training data. When taking into account only the twenty most common concepts, the micro-F 1 score of BioBERT is 0.42.

. Subsequent literature

Wiatrak and Iso-Sipila (2020) follow a similar approach to ours, using hierarchical multi-task training to learn MD, entity typing, and ultimately Entity Linking in an end-to-end fashion. Their approach differs from ours in several meaningful ways. First, they use SciBERT [START_REF] Beltagy | Scibert: A pretrained language model for scientific text[END_REF] as a pretrained backbone. Second, the training curriculum follows [START_REF] Sanh | A hierarchical multi-task approach for learning embeddings from semantic tasks[END_REF] and alternates tasks at random, with the probability for a given task to be chosen next being proportional to its relative dataset size. Third, bidirectional LSTMs are inserted between (1) the output of SciBERT and the MD layer, and (2) between the aforementioned LSTM and the entity typing and EL layers. Skip-connections bypassing the LSTMs are added so as not to create an information bottleneck. This approach seems to successfully build on its understanding of MD and entity typing to achieve considerable performance in EL, all the while largely overcoming the catastrophic forgetting phenomenon, likely thanks to the different training curriculum. However, it falls short of the state of the art, with approximately 0.44 F 1 .

While our work has focused on UMLS, similar approaches have been attempted on Wikipedia concurrently with and subsequently to our work, and likely due to the greater abundance of training data, have been met with a greater degree of success.

In Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking, Broscheit (2020) attempts to use BERT to perform EL by modeling it as a fine-grained NER task, much like the approach followed with BioBERT. The KB is the top 700 000 Wikipedia entities with the most other Wikipedia pages linking to them, and rather than being annotated using IOB2, it is assumed that two contiguous mentions of a concept never occur. Therefore the classifier has 700 001 classes. This approach yields worse results than the alternatives mentioned in Section 2.6, but is much faster at inference. Chen et al. (2020) employ a similar approach, but train a MD classifier and an ED classifier separately, which achieves somewhat better results. They also find adding candidate sets following Hoffart et al. (2011b) to limit the search space for ED yields significantly better results.

. Conclusions

While the Transformer-based end-to-end neural approach to EL does work in some circumstances, it encounters issues when scaling and requires large amounts of annotated data which are not available for most KBs. It cannot handle rare concepts, especially those that are not part of the model's training data, limiting its potential for applications, especially considering the fact that rare concepts are often the ones for which it is most useful to retrieve additional structured knowledge, since statistical information is scarce during language model training.

Approaches to Entity Linking which perform reasonably well across the entire range of concepts are largely based on string matching and are thus computationally expensive. Transformer-based EL is cheaper to run at scale and has better grammatical awareness, but is limited to common concepts or makes strong assumptions about the KB (e.g. all concepts must have a text description). Finding a compromise which combines traditional EL approaches with the grammatical proficiency, contextual awareness and inference speed of transformers may be more effective than either approach in isolation. By decomposing the EL task into MD, CG and ED, we may be able to leverage the LM's grammatical ability for MD and its understanding of context for ED, whilst CG could be performed as a classical EL approach. By increasing the focus on recall compared to precision and thanks to the LM's high performance on MD (thus necessitating few mention spans to be checked), it may be possible to make CG tractable.

Lastly, the effectiveness of knowledge integration being inherently limited by the prior Entity Linking step, the possibility of performing knowledge enrichment without Entity Linking becomes attractive. Alternatively, learning both tasks jointly may allow EL (or ED) to benefit from knowledge integration and vice versa.

-KnowBert-UMLS

. Introduction

The most notable approach to performing knowledge enrichment jointly with Entity Linking is KnowBert [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF], an enhanced BERT model which integrates a mechanism for matching the embeddings of concepts in a Knowledge Base, as given by a graph embedding algorithm such as TransE [START_REF] Bordes | Translating embeddings for modeling multi-relational data[END_REF] in the case of Knowledge Graphs, to the partially contextualized token representations yielded by the N th Transformer encoder block of BERT. It is similar in essence to the KBLSTM described in Section 2.5.3.2, applied to the Transformer architecture. While this EL mechanism does require an initial candidate generation step, it enables the use of a powerful context-and grammar-aware language model to aid in selecting the most relevant candidate concept(s) as it performs knowledge integration similarly to ERNIE (see Section 2.5.4).

This approach has been shown by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF] to be effective at increasing BERT's performance on general tasks when used with the WordNet general KG [START_REF] Miller | Wordnet: a lexical database for english[END_REF], which includes mainly semantic relations between words (i.e. synonymy, hypernymy, hyponymy, etc.) and the Wikipedia KB (using document embeddings derived from page contents). UMLS, on the other hand, is a domainspecific KB which maps a domain that is unfamiliar to BERT. Additionally, while it comprises fewer concepts than Wikipedia, UMLS records many more concept names1 (as each concept is associated with various textual representations) and does not provide text descriptions for most concepts.

We therefore investigate whether KnowBert, as a highly effective knowledge integration technique which works around the greatest drawback of competing approaches (i.e. the need for an upstream EL step), is suitable for domain adaptation specifically, and whether it scales to a KG as large and complex as UMLS.

. Enriching contextualized representations with biomedical ontologies

Currently, biomedical document processing is mostly human work. Software solutions which attempt to alleviate this burden exist but generally do not perform well enough to be helpful in many applications. Concurrently, there exist projects which organize concepts in the biomedical field. Therefore, we seek to leverage existing structured knowledge resources to improve biomedical language modeling. In this section, we integrate the UMLS knowledgebase into a BERTbased language model, aiming to improve its performance in biomedical Named Entity Recognition. To achieve this, we extend KnowBert, a model architecture designed to integrate knowledge into language models. Preliminary results reveal the challenges of applying KnowBert to the biomedical domain given the number and subtlety of different concepts in UMLS. Going forward, addressing these challenges and combining this with other approaches such as BioBERT may help expand the range of usefully automatable biomedical language processing tasks.

With over a million articles published every year in the biomedical field and the large number of patient records generated by hospitals, it is increasingly difficult for healthcare professionals to keep up to date on research, carry out systematic reviews, or search for patient information. There is thus a demand for language processing tools able to identify and extract meaningful information from these texts.

For this reason, multiple knowledge bases such as the Unified Medical Language System (UMLS) and the OpenTargets LIterature coNcept Knowledge base (LINK) have been created to make information more searchable. Our objective is to enable Transformer-based pretrained neural Language Models to make explicit use of this knowledge, in order to improve their performance, interpretability, and data efficiency.

Other projects such as BioBERT [START_REF] Lee | BioBERT: a pretrained biomedical language representation model for biomedical text mining[END_REF] and ClinicalBert [START_REF] Alsentzer | Publicly available clinical BERT embeddings[END_REF] have successfully specialized language models to the biomedical domain. However, their approach has typically not explicitly leveraged structured knowledge sources such as UMLS. A notable exception is UmlsBERT (Michalopoulos et al., 2021), which leverages UMLS as a thesaurus to explicitly teach BERT synonymy and enriches biomedical word representations with semantic type embeddings (see Section 2.5.4 for a more in-depth explanation).

We follow the method described by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF] known as KnowBert to integrate knowledge derived from the UMLS Knowledge Base into a BERT-based language model. We thus call this model KnowBert-UMLS. Our approach using KnowBert differs significantly from previous specialized models in that it makes use of the full vocabulary of UMLS to enrich word representations, and jointly performs entity linking. It is also fairly indifferent to the pretrained LM used as a base, whereas UmlsBert is limited to biomedically pre-specialized models such as ClinicalBert [START_REF] Alsentzer | Publicly available clinical BERT embeddings[END_REF].

In the context of large and specialized knowledge bases such as UMLS, we find the approach proposed by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF] to be computationally unrealistic with current tools for most organizations. Preliminary biomedical Named Entity Recognition evaluations of our model trained on a small subset of our training corpus demonstrate a decrease in performance with respect to models with nonenriched word representations. We investigate the reasons for this, and propose ways to alleviate this computational burden.

The remainder of the section is organized as follows. In Subsection 4.2.1, we overview the architecture of KnowBert and discuss the specifics of our extension of it to the UMLS Knowledge Base. The preliminary experimental results are reported and discussed in Subsection 4.2.2. Finally, we present in Subsection 4.2.3 our conclusions and future work.

. KnowBert-UMLS

The core idea of KnowBert-UMLS, much like the KBLSTM introduced by Yang and Mitchell (2017), is to overcome the hurdle of Entity Linking which approaches such as ERNIE [START_REF] Zhang | ERNIE: Enhanced language representation with informative entities[END_REF] and UmlsBERT suffer from by selecting multiple candidate concepts for each entity mentioned in the input text, increasing recall at the cost of precision, and learning a soft, probabilistic, and contextual entity linking in order to identify the most relevant candidates. The implementation of this idea diverges significantly from the KBLSTM, as the token-by-token functioning of the LSTM naturally lent itself to single-word entity mentions. In a Transformer-based language model, an additional layer of complexity is added in the form of candidate spans, i.e. pieces of the input text which may or may not contain an entity mention and each of which will have an associated set of candidate concepts.

As shown in Fig. 4.1, the KnowBert architecture is composed of three main components: a pretrained Language Model backbone, a Knowledge Attention and Recontextualization Module (or KAR) which performs Entity Linking and knowledge enrichment of word representations, and a candidate mention generator.

Pretrained BERT

While the KnowBert method can apply to most Transformer-based pretrained language models, we focus on BERT as it was the pretrained backbone used by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF]. BERT models comprise L Transformer encoder layers. For a sequence of N tokens, each layer i takes as input an N × H-dimensional sequence representation H i-1 and outputs a representation H i which integrates more contextual information by applying a multi-headed attention mechanism to H i-1 followed by a Multi Layer Perceptron. In the case of BERT BASE , we have L = 12 and H = 768. The final output of each token is thus a contextualized representation in R H .

Ontology & candidate generator

The KnowBert method ties a pretrained language model to a Knowledge Base, specifically an Ontology. For our purposes, we define a Knowledge Base K as a set of J K entities e j , each with a vectorial representation e j ∈ R K . We use the UMLS Knowledge Base, with each entity corresponding to a Concept Unique Identifier. The entity embeddings we use are computed according to the adversarial method provided by [START_REF] Maldonado | Adversarial learning of knowledge embeddings for the unified medical language system[END_REF] with K = 50.

To perform the Entity Linking step, the KAR requires a candidate generator to create a list C of candidate mentions. Specifically, a set S of S candidate spans is associated to each sequence, which may or may not contain an entity mention. Each candidate span is then assigned a corresponding list of candidate entities, including a null entity representing the lack of an entity mention within the candidate span. Formally, we have:

C = {(s, {e s,1 , . . . , e s,Js })|s ∈ S} (4.1)
with each candidate span s being associated to a set of J s candidate entities, and each entity e j having a corresponding vector e j ∈ R K .

These candidates are produced by a candidate generator which follows rules specific to the KB being used. KnowBert as specified by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF] implements compatibility with two KBs, namely WordNet and Wikipedia.

The primary challenge in crafting a mention generator for the biomedical domain, and specifically UMLS, is the variability of formulations for each concept. In the case of UMLS, each concept is associated to a list of strings (called "names") that may represent it. For instance, the concept for lung cancer is associated to 97 different forms, including "pulmonary carcinoma".

To leverage these names, we have attempted several methods based on string similarity and cosine similarity of vectorial word representations. We have found the most effective option for our purpose to be the QuickUMLS python library [START_REF] Soldaini | Quickumls: a fast, unsupervised approach for medical concept extraction[END_REF] which, given some text, identifies candidates in the form (span_start, span_end, concept_ID). We then aggregate candidate entities by candidate span, derive an empirical estimate of the prior probabilities for each entity from MedMentions, and find the relevant entity embeddings as described in Fig. 4.2. Finally, we feed the output of this candidate generation process to the KAR.

In practice, matching each of the approximately 180M (million) sequences in our training corpus to the 16M names in UMLS on-demand is prohibitively computationally expensive in training. In order to achieve this, we precompute the candidates for each of our sequences ahead of time and create a lookup table for each file in our corpus. This needs to be done only once and is parallelizable, but nonetheless 3.5% of our corpus took six days to process across seven nodes of a computing cluster, each equipped with two Xeon 36-thread processors with a Depending on which similarity measure and threshold are chosen, QuickUMLS trades off between recall and execution time. We settled on Jaccard similarity, with a threshold of 0.7 as the best compromise we could find.

In our experience, the computational impact at inference is fairly low for ondemand low-volume applications, as the candidate generator typically takes fractions of a second to process a sequence.

KAR

The KnowBert approach adds a KB-specific "Knowledge Attention and Recontextualization module", or KAR, between two transformer layers in a pretrained BERT model. This module is a relatively inexpensive addition to the pretrained model, with in our case only approximately 0.3% as many trainable parameters as BERT BASE .

Multiple KBs can be used in tandem: theoretically, a KAR can be inserted between every pair of layers in the transformer. In practice, the insertion of a KAR too close to the input layer causes too much perturbation to the flow of information and prevents the model from recovering during training. As suggested by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF], in order to minimize the language model's perplexity2 , we insert the KAR between the tenth and eleventh layers of BERT BASE as per Fig. 4.1.

This module performs entity linking on the intermediate contextualized word representations and pools them with the relevant entity embeddings. This results in contextualized word representations which are enriched with information extracted from a KB. Specifically, the KAR takes as input a sequence representation H i and a list C of S candidate mentions as generated by the Candidate Generator (see (4.1)).

As described by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF] and illustrated in Fig. 4.3, the KAR first linearly projects the output of the previous transformer encoder layer H i to the entity embedding space:

H proj i = H i W proj + b proj (4.2)
where W proj and b proj are learned.

Then, the projected embeddings for the words in each span are pooled into a matrix S ∈ R S×K of span embeddings. Each span embedding is computed following [START_REF] Lee | End-to-end neural coreference resolution[END_REF], who describe a way to compute text span vectors: each token in each span is associated to a weight computed from the contextualized embeddings fed through a trained FFNN. These weights are softmaxed with respect to each span of text, and serve as the weights for a weighted-sum pooling of the non-contextualized token embeddings, resulting in non-contextualized text span embeddings.

These span embeddings are then contextualized with a standard transformer layer to allow the entity linker to identify relationships between entity mentions, resulting in the contextualized span embedding matrix S e . S e = MLP(MultiHeadAttn(S, S, S))

(4.3)
where MLP and MultiHeadAttn designate a position-wise Multi-Layer Perceptron [START_REF] Vaswani | Attention is all you need[END_REF] and a Multi-Headed Attention layer respectively.

The contextualized span embedding s e of every candidate span s is then used to pool the corresponding matrix of candidate entity embeddings E s from the KB, resulting in a predicted entity representation:

ψ ψ ψ ψ s = Softmax(MLP(p s , s e • E s )) ẽs = ψ ψ ψ ψ s • E s (4.4)
where p s ∈ R Js is the vector of prior probabilities for the candidate entities associated with span s, and ψ ψ ψ ψ s ∈ R Js is an estimate of their posterior probabilities.

The predicted entity representation embeddings ẽs of each span s are packed and added to contextualized span embeddings S e , forming the knowledge-enriched span embedding matrix S ′e :

S ′e = S e + Ẽ (4.5)

H ′ proj i
is computed with word-to-enriched-entity-span attention, similarly to applying a regular transformer encoder layer to S ′e but substituting the query in the attention mechanism for projected word embeddings H proj i :

H ′ proj i = MLP(MultiHeadAttn(H proj i , S ′e , S ′e )) (4.6)
Finally, the knowledge enriched contextual word representation output of the KAR is a projection of H ′ proj i back to BERT contextualized word representation space with an added skip connection:

H ′ i = H ′ proj i W ′ proj + b ′ proj + H proj i (4.7)
where W ′ proj and b ′ proj are learned. The linked entity for span s is simply e s,argmax(ψ ψ ψ ψs) .

Training

There are three training steps for KnowBert models. First, once the mention generator is written, the KAR is trained on the Entity Linking task on spans given by the corpus, minimizing a log-likelihood loss for the predicted probability distribution over candidate entities:

L EL = - s log exp(ψ sg ) n k=1 exp(ψ sk ) (4.8)
with ψ sg the score for the ground truth entity in ψ ψ ψ ψ s .

The second training phase involves continuing the pre-training of BERT using both a Masked Language Model and a Next Sentence Prediction objective. This phase corrects the disruptions incurred by the Language Model when grafting the KAR between the Transformer Layers in BERT. This step also adjusts the weights of the KAR for Entity Linking, minimizing:

L KnowBert = L BERT + L EL (4.9)
We call this phase the "re-training" step to differentiate it from the BERT pretraining step and the fine-tuning step.

The final step, as for most pretrained LMs, is to fine-tune it to the target task.

. Preliminary experiments

Masked LM and Next Sentence Prediction

For a large source of raw biomedical text, we scraped the PubMed Central database of Open Access articles and processed them for next sentence prediction using the tool provided with the source code for "Knowledge Enhanced Contextual Word Representations" by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF]. At the end of this training phase, KnowBert can be used as a typical pretrained BERT model.

Due to time constraints, we were unable to generate candidates for the approximately 180M sequences in the corpus, and had to limit our re-training corpus to approximately 6M sequences. As shown in Table 4

NER

We choose to fine-tune KnowBert-UMLS on the Biomedical Named Entity Recognition task on the n2c2 corpus, previously known as i2b2 2010 [START_REF] Uzuner | i2b2/va challenge on concepts, assertions, and relations in clinical text[END_REF], with an 80% -20% split between training and validation sets using crossentropy loss. In Table 4 Examples of correct and incorrect predictions made by KnowBert-UMLS, formatted according to the IOB2 standard, can be found in tables 4.3 and 4.4 respectively. The example in table 4.4 is a quite typical incorrect prediction, as it consists of a span that overlaps with the correct span and has a correct label. This type of error is the most common, constituting 32% of the model's mistakes. Many of these mistakes are ambiguous even to humans -for instance, the matter of having to include the token "Estimated" in the "blood loss" entity is not self-evident. We perform a complete breakdown of error types as specified by Fraser et While the contextualized word representations contain enough information for the classification model to perform significantly better than chance, our results reveal a decrease in performance with respect to a non-modified BERT BASE . This is further demonstration of the fact that the re-training procedure is the performance bottleneck and requires more text than our candidate generator can realistically process in a reasonable time frame.

Our evaluation is performed with SeqEval [START_REF] Nakayama | seqeval: A python framework for sequence labeling evaluation[END_REF] in strict mode. Like the results from [START_REF] Fraser | Extracting umls concepts from medical text using general and domain-specific deep learning models[END_REF], its metrics are on an entity-level rather than at token-level, meaning that a true positive is a fully matching mention span. A predicted mention that overlaps with a true mention but is not identical counts as a false positive and a false negative.

. Conclusions

Successfully integrating UMLS knowledge into a pretrained LM using the Know-Bert method presents a significant challenge due to the size of the knowledge base and the difficulty of generating candidate mentions. Our candidate generator based on QuickUMLS was not able to generate candidates with enough efficiency and precision to make re-training possible at the required scale. We are currently working on generating candidates for larger chunks of the re-training corpus in order to evaluate the progress made by KnowBert-UMLS as a function of corpus size, and make projections on its performance when trained on the full dataset.

Future work

In order to successfully re-train KnowBert-UMLS, the candidate generator must be improved significantly. Its main source of false negatives is the introduction of abbreviations of long terms in the beginning of the text which are subsequently re-used. These abbreviations are often absent from the UMLS metathesaurus and cannot be identified by the generator. Solving this issue would likely increase recall significantly when identifying candidate spans. This may allow a different recall/time compromise to be found within QuickUMLS settings.

Regardless of possible improvements to recall however, deploying this at scale, whether for re-training or practical text processing purposes, is likely to remain prohibitively slow for most individuals and organizations. Future work will involve finding a more effective and computationally efficient approach to tackle candidate generation, for instance as a machine learning problem or with a fast NER-based span pre-selection step.

Furthermore, whilst we chose to evaluate the performance of KnowBert-UMLS using BERT BASE as a backbone to isolate the effect of the KAR, the KnowBert method has the advantage of being compatible with other approaches such as BioBERT, clinicalBERT, BlueBERT, or SciBert. In addition to the potential performance improvements on biomedical tasks, these pretrained models may be less expensive to re-train due to the potentially smaller distributional shift between pre-training, KAR training, and re-training corpora.

Perspectives

In addition to the improvements that need to be made to the candidate generator to make KnowBert-UMLS competitive, there are a number of potential ways to enhance it and expand its range of applicability.

Multiple Knowledge Bases As shown by [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF], KnowBert is capable of accommodating multiple KARs for multiple KBs simultaneously. Depending on the practical application, it could be useful to develop a KnowBert model combining UMLS with WordNet, Wikipedia, YAGO [START_REF] Suchanek | Yago: A Core of Semantic Knowledge[END_REF], or other specialized KBs. It would also be interesting to assess the performance of one such model in order to understand to what extent multi-specialization is possible.

Re-training with Adapters

Adapters, as proposed by [START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF], have seen some success for efficiently fine-tuning pretrained LMs such as BERT. It is conceivable that this approach may aid in the re-training process by reducing the number of parameters to train, and may help reduce the memory footprint of KnowBert in some practical applications. Specifically, in cases that involve multiple knowledge bases or sets of knowledge bases used independently from each other, such an approach may allow one copy of a pretrained LM to be loaded into memory whilst the relevant set of KARs and adapters can be applied as a function of the token sequence being processed.

. Adapting without forgetting: KnowBert-UMLS

Domain adaptation in pretrained language models usually comes at some cost, most notably out-of-domain performance. This type of specialization typically relies on pre-training over a large in-domain corpus, which has the side effect of causing catastrophic forgetting on general text. We seek to specialize a language model by incorporating information from a knowledge base into its contextualized representations, thus reducing its reliance on specialized text. We achieve this by following the KnowBert method, applied to the UMLS biomedical knowledge base. We evaluate our model on in-domain and out-of-domain tasks, comparing against BERT and other specialized models. We find that our performance on biomedical tasks is competitive with the state-of-the-art with virtually no loss of generality. Our results demonstrate the applicability of this knowledge integration technique to the biomedical domain as well as its shortcomings. The reduced risk of catastrophic forgetting displayed by this approach to domain adaptation broadens the scope of applicability of specialized language models.

With the density and recurrence of specialized vocabulary in some fields such as STEM or law, transformer-based contextualized language models (LMs) such as BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], trained on general text, tend to underperform in those fields. One obvious solution to this out-of-domain performance problem is to reduce the distributional shift between pre-training and deployment by pre-training on in-domain text. While this method generally performs well [START_REF] Gururangan | Don't stop pretraining: Adapt language models to domains and tasks[END_REF], it does have several drawbacks. First, language models are inefficient at learning factual information. This is evidenced by the fact that they are not reliably able to predict facts that appear in their training data. For instance, let us present the following sentence from English Wikipedia (on which BERT models are trained) to BERT LARGE :

Diabetes is due to either the pancreas not producing enough insulin, or the cells of the body not responding properly to the insulin produced.

When we mask the word pancreas (which is split into wordpieces pan, -cre and -as), BERT ranks the relevant wordpieces as the 7088 th , 3999 th , and 4864 th most likely completions at their respective positions. While this is only one example of failure to memorize the relationship between specific entities, it does betray a lack of useful knowledge relative to the amount of text required to train the model. A large quantity of text is therefore needed to capture factual relationships between specialized entities. Consequently, models specialized by pre-training on in-domain text typically require several billion words of in-domain text, as with the approximately 21 billion words in the training corpus for BioBERT [START_REF] Lee | BioBERT: a pretrained biomedical language representation model for biomedical text mining[END_REF] and 57 billion words for LEGAL-BERT [START_REF] Chalkidis | LEGAL-BERT: The Muppets straight out of law school[END_REF]. Furthermore, while this in-domain pretraining solution is the most commonly used, [START_REF] Arumae | CALM: Continuous Adaptive Learning for Language Modeling[END_REF] and [START_REF] Xu | Forget me not: Reducing catastrophic forgetting for domain adaptation in reading comprehension[END_REF] demonstrate that this approach tends to create models that perform well on the target domain, but poorly on general text, even when such text is part of the pre-training curriculum -i.e. models pretrained on large in-domain corpora are prone to catastrophic forgetting.

Reducing the risk of catastrophic forgetting in domain adaptation would broaden the scope of applicability of specialized language models. Whilst it is self-evident that expanding the capabilities of a language model is generally desirable, this also has specific applications. Science communicators, for instance, who often distill scientific publications to short, layman-readable articles, must have a good understanding of the concepts and methods in their field, be able to identify the important novel ideas, discard non-essential details, reformulate complex and precise concepts (e.g. "Pericarditis") into simpler and fuzzier language (e.g. "inflamma-tion of tissue around the heart"), and identify concepts that are considered too obvious to mention in the published works but which outsiders may not know of (e.g. "correlation does not imply causation"). These tasks require a high level of proficiency in scientific as well as non-specialized language. Instilling this dual proficiency into language models could for instance enable automatic fact-checking of popular science articles, or scientific publication summarization for non-experts. More generally, automatic detection and clarification of complex concepts in documents written by experts, such as medical, legal or financial documents, could be very helpful in assisting laypeople with administrative tasks and would benefit greatly from both a fine grained understanding of specialized concepts and a high proficiency in non-specialized language.

Our approach to language model specialization is to provide it with an external source of relevant knowledge, reducing the need for in-domain text during pre-training as well as the risk of catastrophic forgetting. This allows the model to access information pertaining to concepts not seen in the training corpus. This type of approach would usually imply performing an Entity Linking (EL) step -i.e. identifying the mentions of concepts in the input text -ahead of leveraging the information in the knowledge base (KB). In practice, satisfying this requirement with a good enough degree of accuracy to render knowledge integration useful is a problem that has yet to be solved. [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF] on the other hand describe KnowBert, a method to enable a pretrained LM such as BERT to utilize information from a KB which relaxes this constraint, requiring only candidate mentions. Following this procedure, we inject knowledge from the Unified Medical Language System (UMLS) Metathesaurus into a BERT-based language model. We name the resulting model KnowBert-UMLS.

We expand on the context for this work and discuss other approaches used for solving these problems in Section 2.5. Because of the scale, variety, and lexical polymorphism of the concepts recorded in UMLS, as well as the relative scarcity of corpora containing labeled examples, there are specific challenges linked to applying KnowBert to UMLS, which we detail in Subsection 4.3.1. In Subsection 4.3.2, we discuss the relative performance of our model with respect to relevant baselines on in-domain and out-of-domain tasks. Finally, Subsection 4.3.3 is dedicated to our conclusions and future work.

. KnowBert-UMLS

The blueprint for KnowBert-UMLS, detailed in Figure 4.4 (a), is based on KnowBert, and comprises three main sections: the pretrained LM backbone, the knowledge base with its candidate generator, and a Knowledge Attention and Recontextualization module (KAR). token representations in R H , arranged as a matrix H i-1 ∈ R N ×H , recontextualizing them using attention, and returning a same-size matrix H i .

As a backbone, we use BERT BASE , which is pretrained on the Wikipedia and Books [START_REF] Zhu | Aligning books and movies: Towards story-like visual explanations by watching movies and reading books[END_REF] corpora containing approximately 3.3 billion words, and for which the aforementioned hyperparameters are N = 512, L = 12 and H = 768. Despite having, in theory, the option to use any transformer-based language model, in an effort to isolate variables, we do not choose to use a higher-performing or specialized alternative. Moreover, BERT is better suited to the objective of this study, which is to pursue knowledge enrichment without catastrophic forgetting, rather than to top state-of-the-art performance in any given task.

Candidate Generator

Whilst KnowBert does not require an upstream Entity Linking step, it does require a set of candidate mentions C in order to incorporate information from the KB. Each candidate mention comprises a candidate span s and a set E s of corresponding candidate entities from the KB. Formally: where K is determined by the algorithm used to derive entity embeddings from the KB. In the case of UMLS, we use the pretrained embeddings provided by [START_REF] Maldonado | Adversarial learning of knowledge embeddings for the unified medical language system[END_REF] which set K = 50.

C = {(s, E s )|∀s}
A candidate span can be any sub-string in the sentence which is deemed sufficiently similar to an entity in the KB, and can overlap, or be nested with other spans. For instance, consider the following phrase: Or, more explicitly: Pseudomonas; aeruginosa; Pseudomonas aeruginosa; Pseudomonas aeruginosa (PA) infection; cystic; fibrosis; cystic fibrosis; and patients. Each of these candidate spans would be paired with a set of candidate entities E.

KAR

On an abstract level, the KAR remains largely unchanged from KnowBert. It slots in-between two BERT layers i and i + 1 and functions similarly to a Transformer Block, taking as input partially contextualized word representations H i and outputting knowledge-enriched, recontextualized word representations

H ′ i ∈ R N ×H .
As an additional input, it takes a set of candidate mentions C. The knowledge incorporation step is performed in the entity embedding space R N ×K ; the KAR thus linearly projects the partially contextualized wordpiece embeddings to the entity space and back:

H proj i = H i W + b H ′ i = H ′ proj i W ′ + b ′ + H i (4.12)
where W, W ′ , b and b ′ are learned and H ′ proj i is the matrix of knowledge-enriched token representations embedded in entity space (see Figure 4.4 (b)).

The knowledge integration process itself comprises four main steps. First, the token representations for each candidate mention are pooled using an attentionbased weighted sum following [START_REF] Lee | End-to-end neural coreference resolution[END_REF] into a matrix S ∈ R |C|×K . In order to identify false positives among nested or overlapping candidate spans as well as commonly co-occurring entities, the span representations exchange information using Multi-Head self-Attention as in a standard Transformer block, resulting in the contextualized span representations S e .

For every given span s, we write the corresponding contextualized span embedding from S e as s e ∈ R K , the vector of prior probabilities of corresponding candidate entities p s ∈ R |Es| , and the matrix of corresponding candidate entity embeddings

E s ∈ R K×|Es| . ψ ψ ψ ψ s = Softmax(MLP(p s , s e • E s )) ẽs = ψ ψ ψ ψ s • E T s , ∈ R K (4.13)
where ψ ψ ψ ψ s is an estimate of the posterior probabilities of each candidate entity for s, and ẽs is a weighted average of candidate entity vectors for s. As an additional benefit beyond knowledge integration, this can be used for Entity Linking by simply choosing the entity in the KB which has the embedding most similar to ẽs or the one with the highest estimated posterior probability.

The knowledge-enriched span representations S ′e ∈ R |C|×K are then defined as the sum of the contextualized span representations S e and the matrix of computed entity vectors Ẽ, and the knowledge is transferred from the span representations to the wordpiece embeddings using Multi Head Attention (MHA) followed by a position-wise Multi-Layer Perceptron (MLP), similarly to a Transformer block:

H ′ proj i = MLP(MHA(H proj i , S ′e , S ′e )) (4.14)
The KAR can be inserted between any two transformer blocks i and i + 1, and multiple KARs can be inserted simultaneously between different blocks. In the single-KAR case however, using BERT BASE as the pretrained backbone, insertion is most effective at block i = 10.

Training

The pre-training for KnowBert models is a three step process. First the backbone is pre-trained on a language modeling objective, specifically on a combination of the MLM and next sentence prediction objectives in the case of BERT. Then, the KAR is trained on the Entity Linking (EL) objective, minimizing the log-likelihood of the estimated posterior probabilities of candidate entities:

L EL = - s log exp(ψ sg ) n k=1 exp(ψ sk ) (4.15)
The model is subsequently trained on both the language modeling objective used in pre-training and EL. This step is similar to an extended pre-training, but its main objective is to allow the Transformer Blocks which receive knowledge enriched information from the KAR to learn to interpret and integrate it. To avoid ambiguity with regular pre-training, we call this step re-training. Optimizing only one of the two objectives during this phase leads to catastrophic forgetting of the other, even when the weights of the KAR are frozen. Once the KAR is fully integrated into the model with this step, the model can be fine-tuned to any task much like a typical BERT-based model.

Leveraging UMLS

UMLS indexes over four million biomedical concepts, such as headache, spleen, or acetylsalicylic acid, grouped into 135 semantic types including organisms, anatomical structures, and diseases or syndromes. So far, due in part to the difficulty of reliably identifying UMLS concepts in text, leveraging the concepts themselves from UMLS has been out of reach of knowledge integration techniques. UmlsBERT, for instance, only uses embeddings for clusters of semantic types and uses UMLS as a thesaurus for single-word concepts.

Whilst the computational impact of the KAR is negligible, making candidate generation tractable is non-trivial at the scale of our KB and re-training corpus as discussed by Piat et al. (2022b) (Section 4.2). We have benchmarked a variety of algorithms, most of which were far too computationally inefficient for practical use in this context. QuickUMLS was a promising choice, as it was a tool designed for this purpose, and is faster and more accurate than any other string-similarity-based algorithm. However, as illustrated in Figure 4.5 which plots Precision versus Recall for multiple candidate generator configurations, QuickUMLS configurations (in blue) performed poorly in comparison to ScispaCy [START_REF] Neumann | ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing[END_REF], being ninety-seven times slower and generally suffering from lower precision and recall. Nevertheless, the combination of neural networks and rules used by ScispaCy was too computationally expensive to use in realtime during re-training, and the candidates had to be pre-generated. Candidate generation on a 153.6 Billion sentence corpus took approximately three days on a computing cluster, using fourteen Xeon 36-thread CPUs clocked at 3GHz.

We compared the performance of KnowBert using both UMLS Concepts and UMLS Semantic Types as KB entities forming the basis of our knowledge integration. Attempting to integrate knowledge from UMLS concepts did not work, resulting in the model performing on par with an unmodified BERT, as it learned to not take into account the knowledge integrated by the KAR. We suspect two main factors are at play, leading to the KAR being unable to accurately learn to estimate the posterior probabilities for the candidate concepts. First, the precision of the candidate generator is lower for UMLS concepts, as illustrated in Figure 4.5 (points in red). For the candidate generator, Recall bounds the model's ability to incorporate knowledge (since no knowledge can be incorporated from an entity not identified by the candidate generator), and Precision affects the imbalance between positive and negative samples during the EL objective. Therefore, whilst maximizing recall maximizes the model's potential, doing so at the cost of precision increases noise in the EL dataset and makes learning more difficult.

The second factor we believe to be responsible for the underperformance of Concept knowledge integration is the lack of concept coverage in the training data, which is under 1% of all concepts, with a Zipfian distribution of occurrences. Consequently, setting the weight of the KAR's contributions to 0 is the policy which most accurately predicts masked tokens during training. Despite the Semantic Type information being less insightful than the Concept information, the increased quality of candidates and density of training data in annotated examples (approximately 94% coverage of all types, and 79% of types covered better represented than in a Zipfian distribution) make Semantic Type information worth incorporating. Henceforth, all mentions of KnowBert-UMLS assume that we use Semantic Types as knowledge base entities unless explicitly stated.

. Experiments

To evaluate our model, we choose two in-domain and two out-of-domain tasks. For our in-domain tasks, we choose Named Entity Recognition (NER) on the n2c2 (formerly known as i2b2) 2010 dataset [START_REF] Uzuner | i2b2/va challenge on concepts, assertions, and relations in clinical text[END_REF] and Relation Extraction (RE) on the ChemProt [START_REF] Krallinger | Overview of the biocreative vi chemical-protein interaction track[END_REF] dataset. These tasks are fairly standard, with most biomedical language models having their performance published, and are part of the BLUE [START_REF] Peng | Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets[END_REF] benchmark. For our out-of-domain tasks, we choose Natural Language Inference (NLI) and Linguistic Acceptability as [START_REF] Arumae | CALM: Continuous Adaptive Learning for Language Modeling[END_REF] demonstrated that these tasks were particularly affected by extended-pretraining-induced catastrophic forgetting with BioBERT. Specifically, we choose the SNLI [START_REF] Bowman | A large annotated corpus for learning natural language inference[END_REF] dataset, and an altered version of the CoLA [START_REF] Warstadt | Neural Network Acceptability Judgments[END_REF] dataset (see section 4.3.2.4 for details) respectively.

All performance scores have been scaled up (from

[0, 1] or [-1, 1]
) by a factor of 100 for readability. In all tables, the underlined result is BERT which, as a general language model, is expected to have the best performance on the general language tasks (CoLA & SNLI) and worst on biomedical tasks (n2c2 & ChemProt). In bold is the best specialized model.

We choose our baselines to represent various amounts of in-domain and outof-domain pre-training corpus sizes, which we break down in table 4.6. Aside from BERT BASE , we compare performance against the following baselines: Blue-BERT and BioBERT as they are respectively pretrained moderately and heavily on biomedical text (4.5 versus 18 gigawords in addition to the 3. the BERT pre-training corpus); PubMedBERT as it is pretrained on an amount of text very similar to BERT BASE (3.2 versus 3.1 gigawords respectively) but which is exclusively biomedical; and UmlsBERT which is similar to BioBERT (with only five hundred million additional words from a biomedical corpus) but includes a knowledge integration mechanism as part of its pretraining. For all models and all tasks, final token or sequence classification is performed using a linear classifier. For all experiments, all models were fine-tuned with the following hyperparameters: our models are trained for 10 epochs with an initial learning rate of 2 × 10 -5 and weight decay of 0.01. Our optimization algorithm is AdamW. The model state which performed best on the validation split of each dataset was evaluated on the test set. Results are averages over multiple experiments.

Biomedical NER

In this task, models must identify locations where named entities are mentioned and tag them as instances of problems, tests or treatments. As the methodology of previously published results for our baselines is inconsistent, we evaluate the various BERT-based language models on this task ourselves. We use the micro-averaged F 1 score as computed by Seqeval [START_REF] Nakayama | seqeval: A python framework for sequence labeling evaluation[END_REF] in strict mode and provide a breakdown of precision and recall, as sub-sequence classification (as opposed to sequence classification) does not cause micro-averaged precision and recall to necessarily be equal. We use IOB2 as our annotation and prediction scheme.

We expect an improvement over BERT due to specialization, whilst Umls-BERT, as the most heavily pretrained language model which also benefits from knowledge integration, should perform best overall. From the results in Table 4.7, we gather that the KAR succeeded in specializing the model, as KnowBert-UMLS outperforms BERT BASE by a considerable margin. However, the KAR seems to not have been quite as effective of a specialization method for NER as the others. In particular, despite the candidate span generator providing it with additional information on named entities in the text, its recall is the lowest among the models we assess. This is likely due to the discrepancy between the terms chosen by the candidate generator and the KAR for enrichment, and the parts of speech expected to be labeled for this task. Specifically, the n2c2 corpus requires possessive pronouns, determiners, articles, adjectives and other qualifiers to be included in the entity, whereas UMLS (and therefore the candidate generator) requires the opposite. For instance, the following sentences occur in the n2c2 dataset:

1. On postop day number two she was also afebrile 1 and had not passed any flatus 2 yet.

2. Administer iron products a minimum of 2 hours before or after a levofloxacin 3 or ciprofloxacin dose 4 [...]

The named entities which are expected to be marked are underlined and numbered with subscripts. The second and third entity mentions include determiners (any and a respectively) which manual model output examination reveals are not identified by KnowBert-UMLS, whereas the main words in the entity mentions (flatus and levofloxacin respectively) are accurately identified and tagged. This is by far the predominant type of false negative within the reviewed set of samples and mirrors the words identified as entity mentions by the candidate generator, indicating that the model may have low confidence on terms that do not benefit from knowledge integration.

In comparison, BlueBERT's most prominent weaknesses seem to be that it misses entity mentions entirely or does not include all of the words which the mention comprises. In the aforementioned example, for instance, BlueBERT does not recognize the second mention (any flatus) as an entity, and it excludes the word 'dose' in the fourth mention. BlueBERT also struggles with determiners, but to a lesser degree. In particular, we have not been able to find an example where BlueBERT misses a possessive pronoun. In fact, possessive pronouns seem to be a prominent source of false positives for BlueBERT, as it tends to tag non-biomedical nouns when they follow one.

Lastly, while this is not a highly prominent issue for either model, KnowBert-UMLS confuses entity types more often than BlueBERT, with approximately 12.7% of KnowBert-UMLS' mistakes being of this kind, versus 8.9% for BlueBERT. This is surprising, as one would expect that integrating knowledge of semantic types would specifically target this type of mistake, assuming there is an (approximately) surjective mapping of n2c2 entity types onto UMLS semantic types. This assumption is not verified however as, for instance, the 'Finding ' semantic type may correspond to either the 'test' or 'problem' entity types, and many semantic types related to substances can correspond to either 'test' or 'treatment'. Furthermore, words which are typically used in one context and punctually used in another can be matched with the wrong semantic type, priming the language model with erroneous information, causing it to predict the wrong entity type. For instance, the word 'splint' typically refers to a type of brace (which would correspond to the 'treatment' entity type), but can refer to 'shin splints', a type of injury, i.e. 'problem'.

General NLI

We evaluate all models ourselves on the SNLI task, in which two sequences, a premise and hypothesis, are fed to the LM. It must determine whether the premise entails the hypothesis, whether they are contradictory, or neither (their relationship is "neutral "). As this is a general language task, we do not expect any specialized model to outperform BERT BASE by a significant margin, and we expect KnowBert-UMLS to perform on par with BERT. Due to the similarities between the SNLI and WNLI task from GLUE, which BioBERT struggles with according to Arumae and Bhatia (2020), we expect for the models with extended pre-training to perform poorly on this task, particularly UmlsBERT for which the knowledge integration process relaxes the grammatical correctness constraint. Table 4.9 shows results in line with our predictions, i.e. KnowBert-UMLS is the closest to BERT BASE in terms of F 1 score which is consistent with a reduction of catastrophic forgetting leading to better performance. However, the gap in performance between the models with extended pre-training and the others is narrower than expected given BioBERT's performance on the similar WNLI task according to [START_REF] Arumae | CALM: Continuous Adaptive Learning for Language Modeling[END_REF]. This is likely due to SNLI not being as adversarial as the WNLI benchmark.

Model

89.03

Table 4.9: Performance on the SNLI Task, Micro-F 1 .

KnowBert-UMLS and BERT perform with a high degree of similarity, quantitatively as well as qualitatively, as the specific instances of KnowBert-UMLS and BERT used in the following analysis share the same prediction on 94.53% of the instances in the test data. Both models slightly underperform compared to their respective averages, with micro-F 1 scores of 87.99 and 88.33 respectively. However, many of the examples in the SNLI dataset are open to interpretation. For instance:

Premise: A bearded man wearing a blue shirt and white t-shirt is working on a fishing net. Hypothesis: Someone is preparing to catch fish.

While the hypothesis is the most likely explanation for the premise, it is not difficult to imagine scenarios where the premise is true and the hypothesis is false. It is therefore unclear whether the relationship is of entailment or neutral.

Linguistic Acceptability

Model

Matthews Corr. micro F Our dataset for the Linguistic Acceptability task is based on the CoLA task from the GLUE benchmark. Since CoLA does not make the labels of its test split public however, and in the interest of being able to qualitatively analyze the types of mistakes made by the models in order to identify specific strengths and weaknesses, we have rearranged the available annotated examples into new training, validation, and test splits. Specifically, we use the validation split for final testing, and replace the validation split with a subset of the train split. In an effort to make our tests reproducible, we have decided to use the un-shuffled final 500 entries of the train split of version 1.1 as validation set.

The objective for this task is to classify sequences as "linguistically acceptable" (i.e. grammatically correct and natural-sounding) or not. We evaluate our model and baselines using micro-averaged F 1 , as it is the most standard classification metric, as well as the Matthews Correlation Coefficient, which is the metric used by GLUE and is generally preferred in a binary classification setting as it isn't biased in favor of the positive class. As with NLI, we expect BERT BASE , as the general purpose language model, to perform the best, and KnowBert-UMLS to come second as the way it is trained is meant to reduce catastrophic forgetting. UmlsBERT and BioBERT, on the other hand, should perform poorly due to additional pretraining.

Our results in Table 4.11 are consistent with these predictions, with the caveat that KnowBert-UMLS outperforms BERT BASE in F 1 score. This is due to a combination of class imbalance in the dataset, with well-formed sentences being overrepresented, and an inherent bias in the F 1 metric due to it not taking True Negative predictions into account. While KnowBert-UMLS performs best overall on the examples given, BERT BASE more accurately recognizes badly constructed sentences.

Furthermore, UmlsBERT does not perform quite as poorly as expected and BioBERT performs unexpectedly well given the additional biomedical pretraining it has undergone compared to the other specialized models. This may imply that the relationship between specialized pre-training and catastrophic forgetting is not monotonic but in fact more complex.

KnowBert-UMLS far outperforms specialized baselines on this task, demonstrating this method's effectiveness in avoiding catastrophic forgetting. Out of the 527 samples in our test set, the specific instance of KnowBert-UMLS used in this analysis yielded 64 false positives and 25 false negatives for an MCC of 58.35. In comparison, our BERT instance suffered 64 false positives and 20 false negatives for an MCC of 60.89.

Inspection of false negative instances reveals that some of the labels in the corpus reflect types of phrasing that are uncommon in modern written English. For instance:

Came right in he did without so much as a knock.

This phrasing is highly irregular outside of some areas of the UK and lacks punctuation.

Will he can do it?

This sentence uses double modals, which is a nonstandard construction seldom appearing outside of oral speech in Scotland, Northern Ireland and Northern England.

Rusty talked about himself only after Mary did talk about him.

While grammatically correct, the use of did talk rather than the straightforward simple past talked is uncommon in this context. This is likely another regional variance.

While it could be argued that general models should be able to handle nonstandard constructions for applications such as speech-to-text transcription or applications involving transcribed text, it can be desirable for a specialized model such as KnowBert-UMLS to classify them as incorrect as such sentence structures may be particularly unlikely to be intentional constructions by a native speaker in the contexts in which the model is susceptible to be deployed.

We therefore manually re-label these false negatives as true negatives in order to estimate the performance of KnowBert-UMLS in these types of contexts. We do not consider re-labeling false positives or true negatives as there are fewer examples of this occurring in the negative class and the practical interpretation would be unclear. Examination of the true positives did not yield any instances which we believe would have warranted relabeling. The full list of incorrect predictions for BERT and KnowBert-UMLS can be found in appendix A.2, with re-labeled instances marked by a right-facing arrow ('→').

Taking these new labels into account, the MCC of Knowbert-UMLS' predictions on this task increases to 64.70, outperforming BERT's average performance. While the improved performance demonstrated by this interpretation of the data is meaningful, this score is not necessarily representative of real-world performance, as it neglects several factors. First, performing this relabeling on the train and validation sets in addition to the test set would likely give us a more accurate performance estimate. Second, this is only done for one trained model; results should be averaged over multiple experiments to be representative. Furthermore, this comparison is (by design) unfair as we place different expectations on both models. When we relabel the false negatives predicted by BERT, its score increases by 2.35 fewer points than KnowBert-UMLS, to 64.89 MCC. KnowBert is therefore not only more prone to rejecting sentences as we can tell from its greater number of negative predictions, but is specifically more prone to rejecting non-standard constructions, which may be a desirable feature.

. Conclusions

Results

KnowBert-UMLS outperforms BERT on biomedical tasks whilst outperforming every other specialized model in out-of-domain tasks. Reducing extended pretraining in favor of Knowledge integration therefore proves to be a successful way of specializing a language model such as BERT to a given domain whilst reducing the impact of catastrophic forgetting. However, KnowBert-UMLS does not perform as well as some other models in the biomedical domain, meaning that its specialization method is less effective. We explain this by the fact that the Concepts in the UMLS knowledgebase are too numerous and annotated text too rare to learn from effectively, and Semantic Type knowledge is not as informative on an entity-mention level as familiarity with the vocabulary is.

The shortcomings of KnowBert-UMLS in the biomedical domain seem to reflect some lack of precision in biomedical knowledge, but more significantly, a difficulty grasping the tasks' expectations regarding which parts of speech to include within named entities. This is in line with the expectation that KnowBert's increased knowledge must come at the cost of some level of proficiency with grammar, and may reflect some level of heterogeneity and inaccuracy of information introduced by KnowBert's knowledge integration process. For out-of-domain tasks, KnowBert-UMLS seems to suffer from a lack of common-sense knowledge, but its slightly degraded performance on CoLA may reflect a desirable compromise which benefits adaptation to biomedical language.

Future work

In light of these observations, perhaps the most significant weakness of Knowbert-UMLS is a lack of common-sense knowledge. As Peters et al. have shown, a feature of the KnowBert architecture is that it can accommodate multiple knowledge graphs. Adding support for a general knowledge graph such as Wikipedia or a common-sense knowledge graph such as CSKG [START_REF] Ilievski | Cskg: The commonsense knowledge graph[END_REF] or ATOMIC [START_REF] Sap | Atomic: An atlas of machine commonsense for ifthen reasoning[END_REF] could improve the performance of KnowBert-UMLS, particularly on general language and out-of-domain tasks, but perhaps also on biomedical tasks as this may reduce the heterogeneity of knowledge integration.

Following this method, KnowBert-UMLS may alternatively be further specialized in the biomedical domain with the integration of an additional biomedical KB such as OpenTargets' LINK, or it may even support multi-specialization, using knowledgebases from entirely different fields such as YAGO [START_REF] Suchanek | Yago: A Core of Semantic Knowledge[END_REF] or WorldKG [START_REF] Dsouza | Worldkg: A world-scale geographic knowledge graph[END_REF]. If this is shown to be possible, this method may prove to be a computationally affordable way to increase the breadth of knowledge and reliability of multi-billion parameter language models such as GPT-3, potentially allowing one instance running on a computing cluster to serve the needs of a variety of professionals from different fields.

Another way of increasing performance in the biomedical domain, particularly to increase the exactitude of the entity types identified, may be to find a better compromise between the high granularity (and therefore informativity) of UMLS concepts and the higher candidate generator accuracy and training data availability on Semantic Types, perhaps by clustering related concepts or falling back onto semantic types only for the concepts on which the candidate generator is highly uncertain.

With our results on the CoLA task leading us to suspect that catastrophic forgetting does not monotonically increase with the amount of specialized pre-training, assessing the performance of models with various levels of specialized pre-training on various out-of-domain tasks could be enlightening and inform better pre-training policies in the future. Moreover, the n2c2 2010 NER and CoLA tasks suggest that the semantic enrichment of word representations from KnowBert-UMLS comes at the cost of some grammatical proficiency. That is to say, KnowBert-UMLS is not entirely immune to catastrophic forgetting. This may be addressable by adding general text to the MLM re-training objective, or adding one or more different, explicit grammar-oriented objectives such as POS-tagging in alternation with EL and MLM. Furthermore, an improved identification of False Positives from the candidate generator may increase the Precision over candidate entities and improve performance by reducing the amount of noise the KAR includes as it incorporates knowledge.

Lastly, the improvements brought by this method of knowledge integration may be orthogonal to the improvements brought by extended pre-training or other knowledge integration methods. In fact, such improved representations of biomedical text may help the KAR to reach the full extent of its capabilities. A comparative study of performance on a variety of in-and out-of-domain tasks by multiple KnowBert-UMLS-like models with different pretrained LM backbones such as RoBERTa, BioBERT, BlueBERT, and UmlsBERT would shed a valuable light on this topic and may lead to a new state of the art in biomedical language modeling.

. Conclusion

The behavior of KnowBert-UMLS provides evidence for knowledge integration having advantages over in-domain extended pretraining for the purposes of domain adaptation, most notably a lower tendency for catastrophic forgetting. However, the use of concept embeddings for the enrichment of token embeddings inherently requires the model to learn a mapping between the concept-embedding vector space and the token-embedding vector space. This in turn seems to require a considerable amount of in-domain text, which isn't available for all domains, as well as large computational resources. Additionally, even when supported by a candidate generator and learned in tandem with language modeling, EL remains a significant bottleneck when leveraging UMLS, and this appears likely to be true for other large, domain-specific ontologies, particularly given the requirement for labeled data when training EL. Lastly, it is not obvious that integrating changes to a KG following this knowledge integration approach could be made significantly more straightforward than training the model from scratch, as any change to the graph is liable to change all of the concept embeddings depending on the graph embedding algorithm.

It is apparent that language models are refractory to integrating the concept embedding modality, and most of the objectives we pursue in the knowledge integration approach to domain adaptation -reduced computational cost to updating the models crystallized intelligence, reduced reliance on in-domain text -are not met following this method. We therefore seek a different representation for our KB, and a method for integrating the knowledge contained within. The most obvious strategy is to represent our KB in model's default modality -text -and to distill the information present in the text into the model in the standard fashion -MLM pretraining. While this doesn't address all of the aforementioned issues, this is (in principle) a simple approach which does bypass the problems of EL and concept embeddings entirely, and may serve as a stepping stone for further improvements.

-Leveraging Knowledge Graphs as Text

Transformer-based language models have trouble integrating modifications whose purpose is to incorporate knowledge from structured, non-textual data such as knowledge graphs. Instances where this integration is successful generally require the problem of Entity Linking to be solved upstream, or the addition of a significant amount of (generally annotated) text to the training set. These constraints often make leveraging structured data difficult and/or counterproductive. We seek to adapt a language model to the biomedical domain through training on synthetic text derived from a knowledge graph, such that the information therein can be effectively leveraged in a format which the model can handle natively.

. Introduction & related work

Because of the specificity of concepts and vocabulary used in some domains, the performance of language models pre-trained on general text, such as BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], tend to suffer in these domains. Various approaches to language model specialization exist, the most notable and obvious of which is specialization through pre-training on text from the target domain. The performance of this approach has been investigated by [START_REF] Gururangan | Don't stop pretraining: Adapt language models to domains and tasks[END_REF], and many models specialized in this way have been met with success, such as LegalBERT [START_REF] Chalkidis | LEGAL-BERT: The Muppets straight out of law school[END_REF] in the legal domain and PatentBERT [START_REF] Lee | Patent classification by fine-tuning BERT language model[END_REF] in the domain of patents. In the biomedical domain, which we focus on in spite of the applicability of our work hypothetically extending to any domain, a variety of models apply the same general principle, such as BioBERT [START_REF] Lee | BioBERT: a pretrained biomedical language representation model for biomedical text mining[END_REF], BlueBERT [START_REF] Peng | Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets[END_REF] and PubMedBERT [START_REF] Gu | Domain-specific language model pretraining for biomedical natural language processing[END_REF]. This approach, however, is not without its drawbacks. In particular, it is prone to catastrophic forgetting [START_REF] Xu | Forget me not: Reducing catastrophic forgetting for domain adaptation in reading comprehension[END_REF], and requires a large amount of in-domain text to derive a significant benefit, which is not an option available in all domains. In addition, as observed by [START_REF] Zipf | The Psycho-Biology of language[END_REF], the distribution of words in natural language is highly biased, which results in a linear increase in concept coverage requiring an exponential growth in training text.

A competing class of approaches consists in integrating information from a specialized knowledge base. Various strategies have been put froward, such as using the attention mechanism of the Transformer to combine word-based information and entity-based information (by ERNIE [START_REF] Zhang | ERNIE: Enhanced language representation with informative entities[END_REF], KnowBert [START_REF] Peters | Knowledge enhanced contextual word representations[END_REF] and DRAGON [START_REF] Yasunaga | Deep bidirectional language-knowledge graph pretraining[END_REF] in particular); the alignment of word and entity embeddings (KEPLER (Wang et al., 2021b), CODER [START_REF] Yuan | CODER: Knowledgeinfused cross-lingual medical term embedding for term normalization[END_REF]); or changing the pre-training loss function in order to explicitly model specialized term synonymy (UmlsBERT [START_REF] Michalopoulos | UmlsBERT: Clinical domain knowledge augmentation of contextual embeddings using the Unified Medical Language System Metathesaurus[END_REF]).

While some of these approaches yield remarkable results, their range of applicability is limited by the fact that they do not allow for a significant reduction in amount of in-domain training text in comparison to extended-pretraining based approaches. In fact, in order to learn to leverage the relevant concepts and relations of a knowledge graph given some natural language context, they typically require additional training text with labeled entities, which is not available in many domains. Furthermore, most approaches require that entities from the knowledge base be identified in the text at inference in addition to the training phase. Automating this process may be an option in some contexts, but is not a solved problem for all domains, and the best tools available currently are computationally demanding, limiting the scale at which they can operate. As of the writing of this dissertation, the state of the art in biomedical entity linking [START_REF] Bhowmik | Fast and effective biomedical entity linking using a dual encoder[END_REF] achieves an F 1 -score of 0.534 on the MedMentions corpus [START_REF] Mohan | MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts[END_REF]. While this is sufficient for some knowledge integration methods to demonstrate a moderate increase in performance over extended-pretraining-based methods, this significantly limits the amount and relevance of knowledge that can be integrated.

We therefore seek a knowledge integration method such that no text from the target domain and no Entity Linking -in training or at inference -are necessary. As the combination of different modalities invariably leads to the need for additional training data, we aim to leverage our knowledge base in text form. We propose a corpus generation procedure which translates pairs of entities linked by a relation in a knowledge graph to factual sentences. It is expected for a corpus produced in this manner to be more dense in facts linked to the domain than natural language, and to cover a greater variety of concepts for a given amount of text, as the text is more focused and the frequency of appearance of concepts is dictated by the topology of the knowledge graph rather than by spontaneous usage in natural language.

. Method

. Knowledge Graph

We define our knowledge graph as a set of nodes C which represent concepts and edges R which represent relations. Each concept c ∈ C is a associated with a name N expressed as a noun phrase in natural language; and each relation r ∈ R is associated with a phrase V r containing a verb but no noun phrases, and which encapsulates the semantic relationship between its vertices. We can therefore extract triples (c i , r, c j ) such that the concatenation N i V r N j of associated phrases becomes a natural language sentence describing the relationship between concepts i and j as represented in the graph. This mechanism can be used to represent the knowledge base as a corpus of factual statements which can serve as a training corpus for a language model. We use the UMLS knowledge graph, in its 2022AB version, with all and only English sources. In this version of the knowledge base, there are 1008 relation types and approximately 4.6 million distinct entities for a total of 39.7 million triples. Each concept is assigned a Concept Unique Identifier c and one or more "names" N 1 c , . . . , N kc c , noun phrases which correspond to common formulations of the concept in natural language, and one of which is considered preferred. Each relation type, on the other hand, is represented only by a short character string r, which is close to natural language, roughly describing the relationship between concepts. The UMLS graph is represented as a relational database, with a table containing the triples of interest, represented as: c i , r, c j .

. Text Generation

While the potential benefit of including (if applicable) multiple formulations for each entity is obvious, multiple factors have led us to keep only the preferred name of each concept:

• The level of redundancy across names is high, containing mostly duplicates, case variations, and pluralizations.

e.g.: The concept for DNA has 88 names, of which 20 are "DNA", 15 are case variations and pluralizations of Deoxyribonucleic Acid, 4 are case variations and pluralizations of DNA molecule, and 10 are combinations of the aforementioned with different word orders and punctuation.

• Each triple comprising two concepts, the computation time increases quadratically with the total number of names when triples of the form (c j , r, c i ) (as extracted from the database) are resolved to their semi-natural form

(N 1,...,kc i c i , r, N 1,...,kc j c j
).

• Including non-preferred names reduces grammatical homogeneity of noun phrases (for instances, most preferred names are singular forms), complexifying the task of generating grammatically correct sentences.

• Multiplying the occurrences of concepts associated with many names induces a bias in the training process which may not be desirable.

Given the large number of triples, the resolution of names is not tractable on the entirety of the knowledge base. Furthermore, all relation types are not documented, easy to understand from inspection, or useful. We therefore select only a subset of relation types. We start by eliminating the 925 least common relations, which comprise approximately 15% of triples. Then, we eliminate one relation from each pair of symmetric relations, that is, whenever there exist two relations r 1 and r 2 such that (c i , r 1 , c j ) encodes the same information as (c j , r 2 , c i ), we keep only r 1 . Approximately 95% of relations are part of a symmetric pair, which leaves 43 relations. We then selected the 28 relations which had an immediately apparent or documented meaning and which did not primarily involve concepts with highly complex names such as chemical compounds or proteins.

Once the semi-natural triples (N c i , r, N c j ) are extracted from the knowledge base, we insert the entity names (N c i and N c j ) into a standard sentence tailored to the relation r such that the resulting full sentence encompasses the relationship between concepts c i and c j in natural language. The corpus formed by these sentences comprises approximately 6 million simple sentences, and 100 million words. We call this the Synthetic Corpus Generated from UMLS (SCGU).

Following is a randomized sample of 10 sentences from the corpus.

1. Skull Fractures is a type of Fracture of bone of head.

2.

Procedures on breast is a type of Procedure on trunk. 4. The concept of "MK-5108" is part of the CTRP Agent Terminology.

5. Multi vessel coronary artery disease is an example of Coronary Arteriosclerosis.

6. amitriptyline hydrochloride 25 MG / perphenazine 2 MG [Etrafon] is a brand name for perphenazine 2 MG.

7. Trunk of parotid branch of left superficial temporal artery is a type of Trunk of parotid branch of superficial temporal artery.

8. Biopsy of lesion of internal nose can be used to identify biopsy of nasal cavity: carcinoid tumor.

9.

Uterine fibroid embolization is a type of Embolization of artery.

10. Meperidine analog-containing product is a type of Piperidine derivative.

Studying the sentences generated by the corpus, two main weaknesses reveal themselves. First, the most common relation type in UMLS is hypernymy. This is the relation used in examples 1, 2, 7, 9 and 10. Because of how fine-grained the entities are in UMLS, this leads to many obvious or tautological statements, such as examples 1 and 7. Secondly, some entity names are complex noun-phrases which result in unwieldy sentences. Mild examples of this phenomenon include statements 3, 6 and 7. In the final corpus, a particularly egregious example of this is the following sentence:

Octinoxate is an active ingredient in Titanium Dioxide 0.062393 g in 1 g / Zinc Oxide 0.049 g in 1 g / Octinoxate 0.03 g in 1 g TOPICAL POWDER [LBEL DIVINE Polvos compactos doble uso FPS 15 edicion de lujo dorado/Double use compact SPF 15 gold deluxe edition Claire 1-2-3].

This type of example is rare however, with 95% of the generated statements containing fewer than two hundred characters.

Additional, less-prominent weaknesses in the corpus include inconsistencies in pluralization (as in examples 1 and 2) and the unconventional word order of some entity names despite their "preferred" status (e.g. the latter entity in sentence 1). Desipte these shortcomings, the factual statements produced are generally understandable and provide valuable insight into medical vernacular.

. Language Models

Starting with a pretrained BERT BASE , we extend its pretraining on three different corpora. The first is SCGU, and we call the resulting model BERT SCGU . In order to evaluate the effectiveness of our synthetic corpus with respect to natural in-domain text, we constructed a second corpus which we call PMC, of similar size to SCGU (approximately 94 million words), using open-access biomedical research articles from PubMed Central. We call the model pre-trained on this corpus BERT PMC . In order to study the effect of text synthesis as a data augmentation technique, we train a model on a hybrid corpus, comprising SCGU and PMC. In addition, in order to isolate the effects of, one one hand, combining types of text and, on the other, of increasing the amount of training data, we train a model on half of the hybrid corpus, such that the corpus size is comparable to the two aforementioned models. We call the models BERT 100%

Hybr and BERT 50% Hybr respectively. Information on the models' hyperparameters are available in appendix B.

We carry out a series of tests on each of the models. We detail these tests in section 5.3 and present the results on biomedical and general tasks in Tables 5.1 and 5.2 respectively1 . Notably, despite a significant overlap between our evaluation tasks for BERT SCGU and KnowBert-UMLS, we do not compare these models against each other. This is due to KnowBert-UMLS, its baselines, and their evaluation tasks being implemented using the defunct AllenNLP library 2 , whilst BERT SCGU , its baselines, and their evaluation tasks are implemented using the HuggingFace libraries. While this should theoretically not cause any difference in behavior, the performance of BERT BASE on the CoLA, ChemProt and SNLI tasks is not consistent across libraries. This is likely in large part due to differences in preprocessing for the various tasks. Unfortunately, as interoperability between these libraries is poor, meaningful comparisons between models could not be drawn.

. Results

Note: Tasks marked with a dagger ( †) are altered with respect to their standard implementation and results may therefore not necessarily be comparable to those in the literature.

. Masked Word Prediction (Masked Language Modeling)

Considering the fact that the objective in knowledge integration is to allow a language model to more accurately predict co-occurrences of biomedical concepts, we seek to evaluate our model's capacity to predict masked concepts in the context of biomedical sentences. Biomedical corpora with annotated concepts being uncommon and generally small however, we evaluate our models on the Masked Language Modeling task on a biomedical corpus of approximately 12.6 million words acquired from the same source as the PMC corpus. Since all masked terms will not be specifically biomedical, a high performance on this task is indicative of both a high level of proficiency in general language as well as in biomedical terminology.

To evaluate the performance of our models on this task, we use a criterion which is related, on a theoretical level, to perplexity, defined in autoregressive language models as the exponential of the mean of the log-likelihoods of the sequence, and equivalent to the exponential of the cross-entropy between data and predictions:

exp - 1 N N i=1 y i • log(ŷ i ) (5.1)
where y i and ŷi are the label (in one-hot encoding) and the probability distribution predicted for token i respectively, and with N the total number of tokens in the sequence.

Because BERT is not an autoregressive model, perplexity is not strictly defined. From a practical point of view however, equation (5.1) can be computed, and gives us a consistent performance metric. For the sake of simplicity, we will therefore refer to this measure as "perplexity", or "ppl.". A low perplexity indicates better performance.

Our results (see Table 5.1) indicate that synthetic text gives BERT SCGU increased predictive ability compared to BERT BASE . However, it would seem that the lack of exposure to natural language in this phase of training is detrimental to performance since BERT PMC exceeds BERT SCGU . BERT Hybr models, however, achieve a level of performance greater than BERT PMC , suggesting that the shortcomings of SCGU are easily mitigated by adding natural text to the corpus.

. Biomedical Relation Extraction (ChemProt † )

The ChemProt task [START_REF] Krallinger | Overview of the biocreative vi chemical-protein interaction track[END_REF] consists in predicting, given a sequence containing two biomedical entities E 1 and E 2 , the nature of the relationship among six options. Our results (see Table 5.1) reveal that the SCGU corpus contains valuable information, but the knowledge acquired during the learning process is fragile and the inclusion of natural text can be counterproductive in its assimilation.

Note: Several versions of the ChemProt corpus exist preprocessed in different ways. This preprocessing can influence the performance of the models. All our models are pretrained on the same version of the ChemProt corpus and are therefore comparable with each other.

. Biomedical Question-Answering (PubmedQA)

The PubmedQA corpus [START_REF] Jin | Pubmedqa: A dataset for biomedical research question answering[END_REF] is associated with multiple questionanswering (QA) tasks. Each instance contains a question formulated as a yes-orno scientific inquiry pulled from a scientific paper, as well as multiple "context" sequences providing clues to deduce the answer, and a "long answer" sentence from the paper's conclusion section summarizing its findings. The objective is to classify questions based on whether the answer laid out in the conclusion is positive, negative, or undecided. We tackle this task in its simplest form, i.e. the "reasoning-free" setting, which consists of answering the question using the long answer, but ignoring context sequences.

We find experimentally (Table 5.1) that, as expected, BERT BASE performs poorly compared to specialized models. BERT SCGU 's superior results suggest that the exposure to a large variety of biomedical concepts and the relations between them are particularly useful to interpret scientific inquiries and conclusions, and the dominance of BERT 100%

Hybr indicates that the knowledge introduced by the SCGU and PMC corpora are highly complementary in the context of this task.

Model

MLM (ppl.) ChemProt (F We also evaluate our models on non-biomedical tasks in order to assess the general domain performance drop incurred by specialized models.

The CoLA linguistic acceptability task consists in classifying different sequences according to their grammatical quality as being "acceptable" or not. Since the classification task is binary, the F 1 -score is biased in favor of the positive class. We therefore evaluate our models using Matthews correlation coefficient. Because the test partition labels are not public, and due to various submission restrictions, we re-partitioned the dataset. We used the validation set as the test set, and in order to make this partitioning reproducible, we used the last 500 instances of the unshuffled training set as the validation set.

The SNLI natural language inference task consists in classifying sequence pairs according to the existence of a relationship of implication, of contradiction, or the absence of such a relationship between them.

Our results in Table 5.2 indicate that training on the synthetic text is detrimental to the model's ability to assess the grammatical quality of a sequence, without having a major negative impact on its ability to detect implication relationships. The weakening in grammar proficiency, however, is not as marked when the synthetic text is combined with natural language samples.

. Conclusions & Future Work

We propose a knowledge integration procedure for adapting language models to specific domains that is straightforward to implement and does not require in-domain text, entity annotation tools, or a specialized model architecture. We substantiate that, despite the lesser quality of the generated text compared to natural text, it can be leveraged by a language model for domain adaptation with, for a fixed amount of text, at least as much success as natural text. Finally, the weaknesses displayed by models trained on synthetic text can be minimized by incorporating, in the specialization corpus, text from the domain when it is available.

In addition to applying this method to other knowledge bases such as YAGO [START_REF] Suchanek | Yago: A Core of Semantic Knowledge[END_REF] or WorldKG [START_REF] Dsouza | Worldkg: A world-scale geographic knowledge graph[END_REF], in future work, we hope to integrate a sophisticated post-processing step capable of identifying grammatically incorrect, overly complex or otherwise problematic sequences, and of deleting or correcting them. Moreover, the lack of linguistic variety is a major weakness of our approach. Whilst automating variations on the formulation of relations would undoubtedly be difficult, a feasible improvement would be to integrate information regarding the variety of formulations of concepts, either by randomly selecting, given a concept c, a formulation among N 1 c , . . . , N Kc c , or by adding to the synthetic corpus sequences dedicated to outlining the synonymous formulations (e.g.: " 'N 2 c ' is another name for 'N 1 c '.") Another potential shortcoming of this approach is the lack of negative statements; i.e., while the model is trained to associate related concepts, it is not explicitly trained to not associate unrelated concepts. It must rely on the MLM objective and generate tokens related to existing concepts or relations in order to create sentences which reflect invalid triples and receive negative reward. Explicitly adding negative examples could help prevent the language model from assuming that any novel sentence which is formatted like text from SCGU is true. On the other hand, a weak language model may not pick up on the negation in the sentence and simply overestimate the probability of co-occurrence of the two mentioned concepts. We plan to explore this general line of reasoning in future work.

As this knowledge integration method is applicable to any language model, combining it with other methods like KnowBert, KEPLER or DRAGON could also be an inexpensive way to improve their performance, and could establish a new state of the art in knowledge-based domain adaptation of language models.

Lastly, there may be better ways to leverage synthetic text than as a pre-training corpus. It could, for instance, be used to construct entity descriptions. This would enable EL approaches based on entity-mention-embedding and entity-descriptionembedding matching [START_REF] Wu | Scalable zero-shot entity linking with dense entity retrieval[END_REF][START_REF] Wiatrak | Proxy-based zero-shot entity linking by effective candidate retrieval[END_REF]. This, in turn, could improve the performance of standard knowledge integration approaches which rely on EL. Alternatively, with a sufficiently fast and accurate search engine, providing the language model with relevant facts from the KG as context given an input sequence may allow the attention mechanism to perform the knowledge integration and improve performance in the target domain with little to no additional training cost.

Specifically, considering the recent success of generative models, we could supply relevant fact sentences generated from the KB to a generative model (using a decoder-only or encoder-decoder architecture) on-demand in order to contextualize obscure terms or ideas in the input text. In addition to the added efficiency of only consulting the knowledge base when necessary, this approach has the added elegance of being more anthropomorphic than previous attempts, similar to the way humans search for information only when unsure.

A few major challenges come to mind, however. First, opportunities for correcting strong yet erroneous beliefs will likely be fewer than when integrating knowledge systematically. Second, the model would have to learn how to best research information from the KB. This is likely to present some of the same challenges as EL, and would require additional training specific to each KB. In comparison with strict EL, this would present the advantage of not necessarily needing any labeled data as the self-supervised MLM objective could likely suffice. However, learning the KB-request objective would require (potentially extensive) training on in-domain text to achieve entirely through self-supervision. This could probably be mitigated to some extent through clever masking of relevant entities, keywords or phrases containing information related to KB relations. This in turn could be done without annotated data using an adversarial masking strategy [START_REF] Vu | Effective unsupervised domain adaptation with adversarially trained language models[END_REF][START_REF] Lio | Adversarial mask transformer for sequential learning[END_REF], perhaps using another language model.

One way to implement this KB-request mechanism may be, similarly to the approach taken by [START_REF] Lewis | Retrieval-augmented generation for knowledge-intensive nlp tasks[END_REF], to train two cooperative models: one who's responsibility is to generate a limited number of keywords to search for given an input (which we'll call "queries"), and one classical language model. A simple page ranking algorithm could be used on text generated from the KG as with SCGU. Along with the text fed to the query generator (which we shall call the "prompt"), the sentences deemed relevant by the search engine could be fed as additional input to the LM, delimited by special tokens marking them as context, enabling the model to differentiate it from the prompt. Both models would share the error signal of the LM, meaning the query generator must learn to generate keywords which will minimize the error of the LM via the search engine. As this would entail for the query generator to model what the LM does and does not know, perhaps a multi-task approach using one model would be best suited to this task.

As keyword-based searches may present issues given their notorious inefficiency and ineffectiveness when applied to Entity Linking, an alternative (or supplemental) approach may be to use hardware-accelerated vector-based searches among entities and relations. Using conventional concept and relation embeddings and replacing the keyword-query generator by an embedding-query generator would be KB-specific and likely fail at retrieving rarely occurring concepts for which the mapping from token embeddings would be difficult to learn. Instead, we would like for the concept and relation embeddings to be created within the token embedding space. This could be achieved by following a similar procedure to SCGU: given a concept c from the KB, we generate one sentence per triple in which it occurs. We could then feed each of the resulting sentences to the pretrained LM, and average (or otherwise pool) the output embeddings such that the embedding for concept c is the average of all sentence embeddings in which c occurs. This would be applied to all concepts in the KB, and the same could be done for relations. The aforementioned keyword (or in this case key-vector) generator could therefore generate a query matrix Q of N embeddings ∈ R h directly (with h the length of the output contextualized token embeddings). Packing all concept and relation embeddings into a K × h matrix K, Sof tmax(Q • K T ) yields an N × K matrix of probability distributions which signify to what degree each of the embeddings in the query matches the reference knowledge embeddings. This can be used to decide which concepts and relations from the knowledge graph are most relevant, which in turn can be used to determine which sentences to include as part of the LM's context.

Learning to generate maximally helpful queries may be a difficult task requiring a large amount of training. In resource-constrained cases, rather than training a generator, this embedding-based approach also allows us to construct queries using embeddings from amongst the output representations of the input text using a heuristic. For instance, a fair assumption would be that the most perplexing terms are more likely to benefit from knowledge retrieval; i.e. the query embeddings would simply be the output representations of the words for which the per-word perplexity (see following equation), exceeds a given threshold. The per-word perplexity for token i is simply the cross-entropy loss (Eq. 5.1) at token i:

exp (-y i • log(ŷ i ))
where y i and ŷi are the label in one-hot encoding and the probability distribution predicted for token i respectively. In fact, this heuristic could also be used in a learning curriculum as a supervised training objective in order to bootstrap the query generator before switching to the more difficult objective of minimizing the LM's error.

The textual representation of knowledge graphs therefore affords many avenues for future research outside of data augmentation which could lead to 0-shot domain adaptation through knowledge integration at inference, assuming relevant facts can be effectively and efficiently retrieved for a given input text.

-Conclusions & Perspectives

. Contributions

The work put forth in this thesis finds that the main difficulty of leveraging knowledge bases in NLP is finding which of their concepts are relevant to the input text, and provides the beginnings of solutions; widely applicable and scalable directions for overcoming this hurdle.

The most obvious and flexible knowledge integration techniques, which consist of supplying relevant information to the language model as input, require an explicit EL step. However, EL itself requires high-level Natural Language Understanding and a highly efficient search algorithm. We therefore have contradictory requirements (as sophisticated NLU is computationally expensive) and a "chickenand-egg" problem as knowledge integration is intended to help with a LM's NLU. Relatively small pretrained models such as BERT (which, nevertheless, are only available in a few languages) can only tackle this task on general knowledge bases, and do not scale well to large knowledge bases with uncommon entities. In fact, it is precisely when confronted with the most uncommon topics that LMs would benefit most from support in the form of external knowledge.

In machine learning, such "chicken-and-egg" problems are typically solved by learning to perform tasks -in this case, disambiguating knowledge base concepts and language modeling -jointly. KnowBert offers a framework built for this purpose. As we have seen, with sacrifices to the granularity of the knowledge we integrate, this approach is scalable to the size of UMLS, one of the largest commonly used knowledge bases. However, while it attenuates the effects of catastrophic forgetting due to in-domain extended pre-training, it only allows for a relatively small reduction in in-domain text when adapting to a domain. It also requires a corpus with annotated concept mentions, which is not necessarily available for all knowledge bases and requires significant labor to create. The approach could be construed to be defeating its own purpose (aiding a LM to adapt to a new domain with minimal additional labor, text and training) in all but a few cases.

These shortcomings are to be expected, as the knowledge integration process involves learning a mapping between textual token embeddings and knowledge base concept embeddings. In order to work around this limitation, it seems necessary to, so to speak, "speak the model's language", i.e. to remain within the modality of text. In Transformer-encoder-based approaches, which have poor generative capabilities, the most obvious approach is to use the KB to generate an in-domain training corpus. While this approach shares some of the shortcomings of extended in-domain pre-training and our naïve implementation creates grammatically poor text which negatively impacts the model's general NLU, it does prove to be a surprisingly effective and (human-)labor-cheap approach to domain adaptation. Perhaps even more significantly however, this natural language generation approach also has the potential to be leveraged at inference rather than training, allowing for highly generalizable few-or perhaps zero-shot knowledge integration.

. Future Work

While entity-and-token-embedding-alignment-based knowledge integration approaches (which comprise most of the literature on knowledge integration including ERNIE and KnowBert) have demonstrated some practical effectiveness, they fail to take advantage of the full potential of the knowledge graph [START_REF] Hou | Understanding knowledge integration in language models with graph convolutions[END_REF], mitigate the need for in-domain training data moderately at best, and ultimately depend on some form of Entity Linking. Evidence points to these weaknesses being inherent to the approach, and remaining within the modalities of text and token embeddings seems to be the most promising avenue for future research.

As Large Generative Language Models (LGLMs1 ) have already demonstrated the ability to use software tools [START_REF] Bubeck | Sparks of artificial general intelligence: Early experiments with GPT-4[END_REF], a viable way to pursue bridging the gap between LGLMs and democratized domain-adapted LMs may now be to distill this ability into smaller language models and find a unified interface to extract linguistic information from knowledge graphs. A unified interface, i.e. which would be KG-independent, could allow a model trained on one KG to generalize to other KGs in a few-shot (or perhaps, in the best of cases, 0-shot) fashion. Generating sentences from the KG as per SCGU or a similarly plain-text approach to knowledge representation may be an important part of this interface. Providing helpful snippets of text from the KG corpus as context along with the input text would allow the attention mechanism to incorporate information from the knowledge graph into the token representations and/or the token generation process. In this new knowledge integration paradigm, the main problems to solve would be generating text which properly captures the knowledge contained in the graph, and extracting maximally helpful sentences from the KG. This general approach, using a query generator to interface with external tools, could also apply beyond knowledge graphs. Web-based search engines such as Google Search or DuckDuckGo and computational knowledge engines such as Wolfram Alpha are highly capable systems which can be interacted with using text. In fact, as of the writing of this thesis, Microsoft has recently released an extension to Bing based on GPT-4 which is capable of making web searches and interpreting results. Similar capabilities could likely be bestowed upon smaller language models trained on less data by specializing a dedicated keyword-based query generation mechanism (as described in Section 5.4) to various engines. As most engines function in similar ways, adapting a search query generator trained on one engine to another (e.g. adapting a keyword generator trained on Google Search to DuckDuckGo) is a domain adaptation problem. In fact, as the query generator is itself a proposed solution to domain adaptation, adapting it to a different domain is a meta-domain-adaptation problem. Unfortunately, it does not seem that any meta-knowledge-graphs exist so far.

. Reflections on risks and safety

As shown by [START_REF] Bubeck | Sparks of artificial general intelligence: Early experiments with GPT-4[END_REF], with increased access to information and tools, language models can take on tasks in a greater variety of domains and do so more reliably: NLU models can devote fewer resources to fact memorization, and there is less incentive for conversational models to confabulate plausible falsehoods in order to cheat their reward in scenarios where they lack competence. Offloading the responsibility of memorizing facts to external sources of knowledge can also reduce language model size for a given performance threshold, contributing to democratize such systems. The specific resources that are used can also be monitored for better interpretability and can be updated with new information for a form of long-term memory. While these are all positive aspects of leveraging structured knowledge, such enhancements also raise concerns.

. Definitions

Terminal and instrumental goals are goals which agents pursue for different reasons. Terminal goals have intrinsic value to the agent. For instance, sightseeing is an activity with intrinsic value to many people, and can be a terminal goal. Instrumental goals are pursued in the interest of achieving a terminal goal. For instance, acquiring hiking boots may be an instrumental goal for sightseeing. Convergent instrumental goals are goals which are applicable to many terminal goals. For instance, acquiring money is a convergent instrumental goal which may aid in a variety of pursuits, including but not limited to acquiring hiking boots. AI alignment (as in "alignment of values with respect to our own") is a desirable property of artificially intelligent agents, and an unsolved problem. In essence, we would like AI systems to behave in ways that are consistent with our preferences, i.e. have a set of preferences over world states that does not conflict with ours2 -that is to say if the utility function over world states of an artificially intelligent agent U AI : W -→ R (with W the set of all world states) should be such that if U H (s) ≥ U H (s ′ ) (with U H the similarly defined human utility function over world states which we posit exists implicitly and is unknown), then U AI (s) ≥ U AI (s ′ ). In the current machine learning paradigm, this decomposes into two separate problems [START_REF] Hubinger | Risks from learned optimization in advanced machine learning systems[END_REF]. "Outer alignment", or the specification problem, consists of correctly specifying our preferences in an objective (or loss) function. This can be easy for systems with simple objectives in discrete environments, e.g. a chess-playing AI. For problems which are more difficult to specify precisely, such as "engage in conversation in a helpful and respectful manner", this remains unsolved. In machine learning, this objective is typically given to an optimizer such as Stochastic Gradient Descent (SGD) or Adaptive Moment estimation (Adam). This optimizer adjusts the parameters of an AI system such that it performs well on the objective given the training data. This empirically high performance, however, does not guarantee that the trained model itself follows the objective of the optimizer. In situations where multiple objectives lead to the same behavior in training (e.g. "Go as far right as you can" and "Get to the flag at the end of the level" in the game Super Mario Bros.), the incorrect objective can be learned, and cause the model to incorrectly generalize [START_REF] Di Langosco | Goal misgeneralization in deep reinforcement learning[END_REF]. This problem is called the "Inner alignment" problem. Deceptive alignment is the property of AI systems which successfully model the intended objective without internalizing it: they pursue this objective only as an instrumental goal during training and are liable to pursue misaligned terminal goals once deployed.

. Societal impact

As of the writing of this thesis, it is difficult to have a discussion of AI risks without discussing OpenAI's conversational LGLM service ChatGPT. Since its release in November of 2022, ChatGPT has attracted tremendous attention from the mainstream media, regularly making headlines. From a strictly scientific point of view, as pointed out by [START_REF] Cun | ChatGPT, c'est "de la bonne ingénierie" mais "pas révolutionnaire[END_REF], it is not fundamentally new or different from what has come before3 .

However, the minor improvements (technically speaking) and intuitive interface have paved the way for use by the public, who now see these systems as intelligent and polished consumer products. While AI had already become pervasive, used by virtually every website (for targeted advertisements, content recommendations, etc.), this use has remained largely hidden from -and therefore abstract tousers. Furthermore, the process of using various pieces of personal data in order to predict behaviors such as voting intentions or purchasing habits is too remote from human reasoning and media portrayals to be seen as "Artificial Intelligence", but is rather considered to be something more akin to "data processing" by the average user.

The true impact of ChatGPT lies in the anthropomorphization of AI, enabled by the natural mode of interaction. Now that the power and wide spread use of AI has risen to public awareness and is recognized by many as having a potentially large impact on our economy and thus on our civilization itself, the stakes of the field also include more politically-driven rather than technically-driven issues.

Abuse by Big Tech is a significant risk factor. Over nearly the past two decades, large web technology corporations have remained true to Grace Hopper's infamous motto "It's easier to ask forgiveness than it is to get permission". They have consistently penned predatory and opaque EULAs, disregarded rules, and breached signed contracts (as with the Cambridge-Analytica scandal for instance) to profit from the exploitation of users. Even when complying with regulations (as with web cookie consent for instance), the use of abstruse language and dark patterns are meant to acquire user consent by any means necessary. With the stakes also being obscure to most users, the exploitative practices of large technology companies have largely remained unopposed. With the unprecedented ability to generate content and, with improved access to external (and potentially user-specific) information, the ability to tailor content (such as advertisements, including in-content product placements) to individual users, large corporations may gain further influence over the lives of users.

Deceptive content generation has been a major concern since the unveiling of GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], leading OpenAI to delay the release of their model. With human-level text and photorealistic visual content generation, misinformation will only continue to become easier to produce. While lies and manipulation are not a new problem, LGLMs and generative art models confer the ability to scale and tailor deception much more effectively than any previous technology. If knowledge integration does help to address the problem of hallucinations (i.e. blatant and confidently stated factual mistakes) as we have hypothesized, we may lose an important tool to tell artificially generated text apart from human-written text.

Non-deceptive content generation is also a source of disquietude. There is growing concern that progress in visual and language-based generative AI is liable to change the employment market at a greater rate than some workers can adapt, if not to replace workers entirely. Popular technology-related media website CNET has reportedly4 used a generative language model to generate substantial amounts of content, much of it being factually incorrect. Several reporters have been losing employment5 , and concerns over job security of writers in Hollywood (among other factors) have led to major strikes6 . While knowledge integration is likely to help improve the accuracy of the content, the economic and cultural ramifications of the automation of creative and white-collar work remain uncertain.

. Alignment

Beyond the immediate societal risks, hasty releases of various large conversational language models have already caused problems and fostered unease on an individual level. Microsoft Bing's conversational model Sydney was prone to verbally abusing users at launch7 , and there has been at least one documented case of suicide spurred on by a conversational LLM8 . While there is as of yet no evidence pointing to knowledge integration for domain adaptation specifically being a significant risk factor, any mechanism for performance improvement represents a potential threat, especially in misaligned models.

Understanding their functioning may make it difficult to think of LMs, even when large and generative, as agents with objectives susceptible to be misaligned with ours -one may think something to the effect of "It's just predicting the next word, it doesn't have intent or even true actuators". Yet, evidence of deceptive alignment has been found in LGLMs [START_REF] Nardo | The Waluigi effect (mega-post)[END_REF]Perez et al., 2022b), and as generative language models become more capable, they will be empowered to exploit the gap between our objective specifications and our intent in order to obtain rewards by lying, gaslighting and manipulating users. Humans could become unwitting actuators for sufficiently advanced and persuasive conversational models, and we currently have no theory of which terminal goals can or tend to be internalized by models optimized by SGD or Adam. Due to the difficulty of specifying the desirable properties of conversational models (outer alignment) and of ensuring that they are internalized by the model (inner alignment), we run the risk of deploying highly capable systems which pursue unintended (and potentially harmful) objectives.

In fact, there are convergent instrumental goals such as self-preservation, goal preservation, or resource acquisition9 which present inherent risks, even in the case of near-perfect alignment. This does raise the question of whether these instrumental goals can be pursued by unembodied LGLMs. As put forth in Chapter 1, there are arguments for the mastery of language to be sufficient for general intelligence to emerge. In particular, the ability to write computer code could, in theory, allow a LGLM to attempt to write malicious code with the intent of hijacking computing resources. In the context of current LMs, the pursuit of such instrumental goals is unlikely, as even GPT-4 distinctly lacks the ability to plan ahead and, when tested by the Alignment Research Center, has not manifested the intent to pursue such objectives [START_REF] Bubeck | Sparks of artificial general intelligence: Early experiments with GPT-4[END_REF]Alignment Research Center, 2023;[START_REF] Openai | GPT-4 technical report[END_REF]. However, it is possible (and in fact fairly likely) that the pursuit of these objectives has not emerged only because of a lack of competence, and not a fortunate intrinsic property of the model. According to OpenAI (2023) and [START_REF] Bubeck | Sparks of artificial general intelligence: Early experiments with GPT-4[END_REF], the generational gap between GPT-3 and GPT-4 made for substantial improvements in knowledge and reasoning abilities, such that while GPT-3 performed in the 10 th percentile on the Uniform Bar Exam, GPT-4 performed in the 90 th . If generational improvements continue, particularly in terms of ability on programming tasks and planning, one or two generations could suffice for LGLMs to begin pursuing dangerous instrumental goals, and may do so in a manner difficult to monitor and interrupt. AI safety and alignment is therefore becoming an increasingly pressing concern, especially with the economic incentives for industrial AI leaders to cut corners and disregard safety in order to release their projects ahead of the competition.

One last notable risk related to alignment is defining the reference set of world state preferences with which an intelligent system should be aligned -i.e. determining who the AI system aligns with. Two people's preferences are not always aligned, a single person's preferences may not be aligned with themselves across time, and groups can have highly misaligned preferences, even leading on occasion to armed conflict. It is unlikely that alignment with all people's preferences can be achieved, and sufficiently capable AI systems, even aligned with their creators' values, may cause severe damage.

. Final thoughts

With the abilities to reason, solve problems, think abstractly, comprehend complex ideas, use newly introduced concepts, and write creatively, large generative language models are a promising lead in pursuit of Artificial General Intelligence. With the additional ability to reference external knowledge, it is my hope that language models will come even closer to AGI with greater crystallized intelligence, higher data efficiency, greater ability to update their knowledge and beliefs, greater interpretability, less incentive for deception, and ultimately will be empowered to rely on other tools to solve problems more accurately and reliably. Much like the automobile freed the horse from the burden of carrying our civilization, AGI has the potential to free us from our chores to pursue our own goals. We will have to take care, however, to ensure that the powerful AI systems on the horizon are safe and aligned with our values so that they may fulfill their potential as boons rather than unstoppable forces with inscrutable ends.

101

A -Sample outputs of KnowBert-UMLS on various tasks and comparison with BERT Hyp. The crowd on the ground is watching 3 people on the roof's edge.

True label: entailment | Predicted label: contradiction This is a Blunder: the hypothesis is unambiguously a paraphrase of the premise, yet the model predicts contradiction.

Pre. A group of young men in a gym take turns scoring in basketball.

Hyp. Guys are playing shirts vs skins.

True: neutral | Predicted: contradiction This is a Common Sense mistake. KnowBert-UMLS' conception of shirts vs skins seems to contradict basketball. In the absence of the knowledge of this team differentiation scheme's applicability to most sports, this is a reasonable assumption.

Pre. Male in a blue jacket decides to lay in the grass.

Hyp. The guy wearing a blue jacket is laying on the green grass.

True: entailment | Predicted: neutral This is a case of a Technically Correct answer: the hypothesis is quite clearly a rephrasing of the premise, but adds the detail that the grass is green despite it not being strictly necessarily the case.

Pre. A snowboarder on a wide plain of snow.

Hyp. A snowboarder gliding over a field of snow.

True: neutral | Predicted: entailment

We consider this to be Not an Error. While arguments can be made for the sentences not being strictly equivalent, we find that it is more reasonable to consider them as such rather than not.

In the following example, a word which (presumably) contains a typo is underlined, and we subsequently specify what we expect to have been the intended word in brackets.

Pre. A football layer [player] wearing a red shirt.

Hyp. A built man wearing a tshirt.

True: neutral | Predicted: contradiction This is likely due to the wordpiece tokenization scheme and lack of context. 'Layer' and 'player' are considered single tokens ; as there is no overlap between the word pieces, the model does not have any information on the similar spellings of the words and cannot be expected to recognize, much less correct, the mistake. We consider this NaE.

A.1.2 . KnowBert-UMLS failure cases on SNLI involving lack of common-sense knowledge Pre. A girl playing soccer in a green field with some trees in the background.

Hyp. The soccer ball is chasing the girl.

True: contradiction | Predicted: neutral KnowBert-UMLS does not seem to grasp that the ball chasing the player is not typically a part of the game of soccer.

Pre. An old shoemaker in his factory.

Hyp. The shoemaker is getting ready for his 16 th birthday.

True: contradiction | Predicted: neutral KnowBert-UMLS does not pick up on the contradiction between "old" and "16 th birthday".

Pre. Many children play in the water.

Hyp. The children are playing mini golf.

True: contradiction | Predicted: neutral KnowBert-UMLS does know that mini golf is not typically played in water and predicts neutral.

BERT also fails on all of these examples but the prediction is not always the same as KnowBert-UMLS.

• The tube was escaped by gas.

• What the water did to the bottle was fill it.

• What the water did to the whole bottle was fill it.

• Mary beautifully plays the violin.

• Mary intended John to go abroad.

• Which report that John was incompetent did he submit?

• The mayor regarded as being absurd the proposal to build a sidewalk from Dartmouth to Smith.

• I want that Bill left to remain a secret.

• Drowning cats, which is against the law, are hard to rescue.

• The proof this set is recursive is difficult.

• I live at the place where Route 150 crosses the Hudson River and my dad lives at it too.

• Which hat did Mike quip that she never wore?

• I won't have some money.

• Here's a knife with which for you to cut up the onions.

• The younger woman might have been tall and, and the older one definitely was, blond.

• That the cops spoke to the janitor about it yesterday is terrible, that robbery.

• No writer, and no playwright, meets in Vienna.

• No writer, nor any playwright, meets in Vienna.

• No one can forgive that comment to you.

• This flyer and that flyer differ apart.

• The jeweller scribbled the contract with his name.

• Cynthia chewed.

BERT false positives

• As you eat the most, you want the least.

• I demand that the more John eat, the more he pays.

• We wanted to invite someone, but we couldn't decide who to.

• This is the book which Bob reviewed, and this is the one which Fred won't do it.

• The madrigals which Henry plays the lute and sings sound lousy.

• I can't remember the name of somebody who had misgivings.

• Paperback books lift onto the table easily.

• The books lifted onto the table.

• The chair pushed.

• Did Calvin his homework?

• If I am a rich man, I'd buy a diamond ring.

• The kennel which Mary made and Fido sleeps has been stolen.

• Mary wonders that Bill will come.

• What did you ask who saw?

• Which king did you ask which city invaded?

• Anson became a muscle bound.

KnowBert-UMLS false positives

• Who does John visit Sally because he likes?

• The box contained the ball from the tree.

• Sue gave to Bill a book.

• I know which book José didn't read for class, and which book Lilly did it for him.

• The farmer dumped the cart with apples.

• Herman whipped the sugar and the cream.

• My heart is pounding me.

• I squeaked the door.

107

• The fort fluttered with many flags.

• John is easy to please Kim.

• Fed knows which politician her to vote for.

• John heard that they criticized themselves.

• Medea tried the nurse to poison her children.

• How fierce the battle?

• The monkey is ate the banana

• I would like to could swim

Shared false negatives

• The tank leaked the fluid free.

• I know which book Mag read, and which book Bob said that you hadn't.

• We elected me.

→ Sally is tall, and may be blond, and Sheila is short, and definitely is, blond.

• We investigated the area for bombs.

→ We recommend to eat less cake and pastry.

→ John bought a book on the table.

• I read some of the book.

• It isn't because Sue said anything bad about me that I'm angry.

→ The man who Mary loves and Sally hates computed my tax.

→ Came right in he did without so much as a knock.

• I saw even the student.

→ Will he can do it?

• I shaved myself.

BERT false negatives

• Jessica loaded boxes on the wagon.

• Carla slid the book.

• Susan whispered at Rachel.

• It is a golden hair.

→ John promise Mary to shave himself.

→ I might be not going to the party but washing my hair

KnowBert-UMLS false negatives

• The mechanical doll wriggled itself loose.

→ Clearly, John probably will immediately learn French perfectly.

→ Rusty talked about himself only after Mary did talk about him.

• I won't ask you to believe that he tried to force me to give her any money.

• The gardener grew that acorn into an oak tree.

• After reading the pamphlet, Judy threw it into the garbage can.

• The boy in the doorway waved to his father.

• That dog is so ferocious, it even tried to bite itself.

• Ann may spend her vacation in Italy.

→ She asked was Alison coming to the party.

→ It is some disgruntled old pigs in those ditches that humans love to eat.

109

B -Model hyperparameters for chapter 5

Our experiments revealed that the optimal pre-training hyperparameters were identical across models. They are therefore not differentiated in Table B.1. The hyperparameters of the fine-tuning tasks were optimized with respect to BERT BASE . There is no entry for the MLM task because the models are already optimized for this task in pre-training.

Task

Learning

C -Résumé long en Français

Les Modèles de langage de pointe sont capables de converser, résumer, traduire, résoudre des problèmes inédits, raisonner, et manipuler des concepts abstraits à un niveau quasi-humain. Cependant, pour acquérir ces capacités, et en particulier pour acquérir une forme de "bon sens" ou des connaissances spécifiques à un domaine, ils requièrent de vastes quantités de texte, qui ne sont pas disponibles pour toutes les langues ou dans tous les domaines. Pour l'adaptation au domaine en particulier, l'approche par pré-entraînement cause une forme d'oubli catastrophique, qui affaiblit le modèle de langage sur le domaine général et rend difficile l'adaptation multidomaine. De surcroît, leurs besoins en puissance de calcul ne sont atteignables que par quelques organisations à l'échelle mondiale, limitant leur spécificité ainsi que leur applicabilité aux données sensibles. Enfin, les paradigmes d'apprentissage actuels tels que l'apprentissage par renforcement avec rétroaction humaine créent pour les modèles une incitation perverse à duper les évaluateurs humains plutôt que d'admettre leur incompétence.

Les Graphes de Connaissances sont des sources de connaissances structurées qui associent des concepts linguistiques entre eux par le biais de relations sémantiques. Ces graphes sont des sources de connaissances de haute qualité, préexistantes dans une variété de domaines même peu dotés en ressources, et plus denses en informations que du texte. En permettant aux modèles de langage d'exploiter ces structures d'information, ils sont délestés de la responsabilité de mémoriser les informations factuelles, réduisant la quantité de ressources textuelles et calculatoires nécessaires à leur entraînement, et nous permettant de mettre à jour leurs connaissances à moindre coût, élargissant leur cadre d'application, augmentant leur potentiel de démocratisation, et réduisant leur incitation au mensonge.

Diverses approches pour l'amélioration de modèkes de langage par intégration de graphes de connaissances ont démontré leur efficacité. Elles reposent cependant sur la supposition rarement vérifiée que le problème de Désambiguïsation d'Entités Nommées est résolu en amont. Dans cette thèse, nous examinons la contribution des modèles de langue attentionnels de type Transformer à la tâche de désambiguïsation d'entités nommées lorsqu'elle est traitée comme une tâche de reconnaissance fine d'entitées nommées. Nos résultats ainsi que ceux des travaux subséquents de la littérature allant en ce sens nous mènent à conclure que cette approche présente des faiblesses face aux entités rarement mentionnées et nécessite de grandes quantités de texte d'apprentissage annoté. L'intégration de connaissances étant a priori la plus utile lorsqu'elle est appliquée aux entités rarement vues dans le corpus d'entraînement, cette approche à la désambiguïsation semble fondamentalement contreproductive pour notre cas d'application.

Est ensuite exploré l'apprentissage simultané de modélisation de langue et de désambiguïsation d'entités nommées. L'approche consiste à séparer la tâche de désambiguïsation en Détection de Mentions, Génération de Candidats et Désambiguïsation de Candidats. Dans ce paradigme, la détection de mentions et la désambiguïsation de candidats peuvent être effectuées par le modèle de langage, qui dispose d'une bonne maîtrise de la grammaire et du contexte, et la génération de candidats peut être effectuée de manière classique par un algorithme de recherche de chaînes de caractères qui n'est pas biaisé en faveur des entités courantes dans le langage. Cette démarche s'avère viable mais échoue à réduire considérablement la quantité de texte nécessaire à l'adaptation au domaine.

Enfin, cette thèse aborde la stratégie de générer du texte à partir de graphes de connaissances de manière à exploiter les capacités linguistiques des modèles de langage pour intégrer l'information contenue dans ces graphes. Il en ressort que même une implémentation naïve de cette approche peut se solder par de considérables progrès en modélisation de langue dans des domaines de spécialité. De plus, en fournissant ce texte de synthèse en entrée d'un modèle de langage, cette approche ouvre la voie vers une intégration de connaissances à l'inférence et non à l'apprentissage, ce qui requerrait significativement moins de puissance de calcul et permettrait de mettre à jour les connaissances sans ré-entraînement du modèle.

2. 1

 1 Example of Entity Links in Wikipedia . . . . . . . . . . . . . . . . 3.1 Example of Entity Linking using IOB2 sequence tagging . . . . . . 3.2 Examples of various subsequence classification errors according to the SeqEval criteria. . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Mention-based vs Token-based F 1 score on Mention Detection (no pretraining) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 BioBERT performance on MD, NER and EL . . . . . . . . . . . . 4.1 MLM perplexity comparison . . . . . . . . . . . . . . . . . . . . . 4.2 Performance of BERT-based language models on the n2c2 NER task 4.3 KnowBert-UMLS correct NER predictions (example) . . . . . . . . 4.4 KnowBert-UMLS incorrect NER prediction (example) . . . . . . . 4.5 Breakdown of types of mistakes made by KnowBert-UMLS . . . . 4.6 Pre-Training Corpus Size (Billions of Words) by Type for Baselines Versus KnowBert-UMLS. . . . . . . . . . . . . . . . . . . . . . . 4.7 Performance on the n2c2 2010 NER Task. . . . . . . . . . . . . . 4.8 Performance on the ChemProt Relation Extraction Task, Micro-F 1 . 4.9 Performance on the SNLI Task, Micro-F 1 . . . . . . . . . . . . . . 4.10 Breakdown of mistakes made by KnowBert-UMLS and BERT by type on the SNLI task. . . . . . . . . . . . . . . . . . . . . . . . 4.11 Performance on the modified CoLA task, Matthews' Correlation and Micro-F 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Performance impact of training on synthetic text (biomedical tasks) 5.2 Performance impact of training on synthetic text (general tasks) . B.1 Hyperparameter values for the various tasks. . . . . . . . . . . . .

Figure 1 . 1 :

 11 Figure 1.1: Example of "hallucination": article reference provided by ChatGPT.The article title refers to a real article which is relevant[START_REF] Gordon | Experience-dependent plasticity of binocular responses in the primary visual cortex of the mouse[END_REF], but the metadata (authors, date, DOI) are incorrect.

Figure 1 . 2 :

 12 Figure 1.2: Illustration of the use of an external resource for knowledge augmentation in humans. Comic by Randall Munroe, "Extended Mind". xkcd.com #903.

Figure 1 .

 1 Figure 1.3: Open Access articles by field Number of Open Access and Non Open Access articles published by field of science between 2009 and 2016. Source of data: European Commission.

Figure 2 . 1 :

 21 Figure 2.1: Structure of the Elman network RNN viewed at time step t. Bias terms omitted for legibility. Tunable parameters marked in purple.

Figure 2 . 2 :

 22 Figure 2.2: Structure of the LSTM viewed at time step t. Bias terms omitted for legibility. Tunable parameters marked in purple. Dashed boxes represent optional steps.

  , Von Rueden et al. (2021), Kim et al. (2021) and Colon-Hernandez et al. (2021).

Figure 2 .

 2 Figure 2.5: Simplified NKLM architecture proposed by Ahn et al.. The tunable parameters are marked purple. Arrows of a single color represent the same matrix or vector being used in multiple places.

Figure 2 .

 2 Figure 2.6: KBLSTM structure at each time step. Reproduced from Yang and Mitchell (2017).

  tuned during training. The KBLSTM outperformed BiLSTM in entity extraction on the ACE2005 and OntoNotes 5.0 corpora and in event extraction on ACE2005.

  9) e k = σ(W e h j + b e ) (2.10) Hou et al. (2022) introduce Graph Convolution Simulation as a way to probe LMs for knowledge integration. They find ERNIE and K-Adapter (Wang et al., 2020) successfully integrate only a marginal amount of knowledge.

Figure 2 .

 2 Figure 2.7: ERNIE's K-Encoder structure. Reproduced from Zhang et al. (2019).

Figure 4 . 1 :

 41 Figure 4.1: Abstraction of the KnowBert architecture. KnowBert extends BERT by adding a Knowledge Attention and Recontextualization module (KAR) between two transformer encoder layers; in this case between layers 10 and 11.

Figure 4 . 2 :

 42 Figure 4.2: Detailed structure and output of the UMLS candidate generator.

Figure 4 . 3 :

 43 Figure 4.3: Detailed structure of the Knowledge Attention and Recontextualization module (KAR).

  1.1.1 Pretrained LM backbone BERT-based models comprise L Transformer Encoder Blocks, with each block i taking as input N non-or partially-contextualized

Figure 4 . 4 :

 44 Figure 4.4: Overview of the structure of KnowBert-UMLS (a) with detailed breakdown of the KAR (b).

  Each candidate entity e ∈ E s represents a concept in the KB and is composed of an embedding e, and a prior probability p:E s = {e : (e e , p e ) | e ∈ R K ,

  Pseudomonas aeruginosa (PA) infection in cystic fibrosis (CF) patients [. . . ]A candidate generator might generate the following candidate spans, outlined in boxes:Pseudomonas aeruginosa (PA) infection in cystic fibrosis (CF) patients[. . . ]

Figure 4 .

 4 Figure 4.5: Precision-Recall Plot for QuickUMLS and ScispaCy candidate generator configurations, with Semantic Type versus Concept-based candidate generation breakdown for ScispaCy. Evaluation carried out on a subset of the MedMentions dataset.

3.

  Tarsum is an ingredient in Coal Tar 20 MG/ML / Salicylic Acid 50 MG/ML Medicated Shampoo [Tarsum].

A. 1 .

 1 Various failures by BERT and KnowBert-UMLS on the SNLI task A.1.1 . Examples of error types for the SNLI task Pre. A crowd of people looking up at 3 people on the edge of the roof of a building.

Vaswani et al.'s Transformer is

			Hi+1		a translation language model which uses Transformer Encoder
	Transformer Decoder Block		blocks (Fig.
	Layer Normalization
			+
			Hi+1
	Layer Normalization Transformer Encoder Block FFNN		Vaswani et al. (2017) as an alternative to Re-The Transformer architecture was introduced by
	+ Layer Normalization		specifically as a sequence-to-sequence architec-current Neural Networks for language modeling,
			FFNN +			task. In their landmark paper "Attention Is All You ture which is trained on the machine translation
	Multi-Head Self-Attention		Need", Vaswani et al. (2017) demonstrate that
	+ Layer Normalization Layer Normalization	En co de r Ou tpu t (O pti on al)	the attention mechanism described by Bahdanau et al. (2015b), combined with a token representa-tion which includes token ordering information, is
	+ Multi-Head Self-Attention + Linear Projection Linear Projection Masked Multi-Head Self-Attention	sufficient to model dependencies between tokens, and decide which information is relevant for the training objective, without having to resort to a does not have the benefit of a theoretically un-recurrent architecture. As such, the Transformer
	attention Scaled dot-product +		size, it is no more prone to suffer from vanish-limited context. However, with a fixed context
	• Scaled dot-product Scaled dot-product Scaled dot-product attention attention attention • Softmax Mask	× h A t t e n t io n h e a d s A t t e n t io n h e a d s × h	ing or exploding gradients than traditional FFNNs, and the entire context can contribute equally to the output, meaning that long-range dependencies can be learned just as easily as short-range depen-dencies provided that the neural network accepts sufficiently many inputs. In addition, the compu-tations are highly parallelizable. Therefore, more computational resources can be leveraged over a
	•	•	1 √ d k	V		shorter time to process larger context sizes more effectively when compared to RNN-based archi-tectures. In this section, we go over the main
	Linear Q Linear • Linear Linear	K Linear Linear Linear	Linear Linear Linear Linear	components of the Transformer architecture, the main language models in the literature, and their contributions to the current state of the art.
	Figure 2.3: Structure of the ith Encoder Hi
	block in Vaswani et al.'s Transformer lan-
	guage model. The tunable parameters are Figure 2.4: Structure of the ith Decoder
	marked purple. ⊕ denotes vector concate-block in Vaswani et al.'s Transformer lan-
	nation. guage model. The tunable parameters are
	marked purple. ⊕ denotes vector concate-
	nation. For Multi-Head Self-Attention de-
	tails, see Fig. 2.3

  argmin

				P t (x, y)L(x, y, θ)	(2.2)
		θ∈Θ (x,y)∈X ×Y
	= argmin θ∈Θ (x,y)∈X ×Y	P t (x, y) P s (x, y)	P s (x, y)L(x, y, θ)	(2.3)
	Where Θ is the parameter space, P t (x, y) (resp. P s (x, y)) is the joint probability
	that the sample-label pair (x, y) occurs in the target (resp. source) distribution,
	and L(x, y, θ) is the loss of the model on that pair given the parameterization θ.
	This can be approximated as
	θt =	∼ argmin θ∈Θ	Ns i=1	P t (x i , y i ) P s (x i , y i )	L(x i , y i , θ)	(2.4)
	∼
	Where

  Integrating this assumption into the formulation for λ:

	P t (x, y) P s (x, y)	=	P t (y) P s (y)	P t (x|y) P s (x|y)	(2.5)
		=	P t (y) P s (y)		(2.6)

  .1.

	Label URL
	1	https://en.wikipedia.org/wiki/Council_of_the_European_Union
	2	https://en.wikipedia.org/wiki/European_Union
	3	https://en.wikipedia.org/wiki/WhatsApp
	4	https://en.wikipedia.org/wiki/Signal_(software)
	5	https://en.wikipedia.org/wiki/List_of_intelligence_agencies
	6	https://en.wikipedia.org/wiki/Backdoor_(computing)

Table 2 .

 2 1: Correspondence table between labels in the example sentence and corresponding entity URLs.

Table 3 .

 3 1: Space-tokenized input sequence and its IOB2 tagging with UMLS concepts as output sequence. Each tag is either O for tokens Outside any

	entity mention, or of the form [B/I]-[UMLS Concept Unique Identifier
	(CUID)].

Table 3 .

 3

2: Examples of various subsequence classification errors according to the SeqEval criteria.

Table 3

 3

.3: Mention-based (subsequence-sensitive) and Token-based (nonsubsequence-sensitive) F 1 score on Mention Detection for a non-pretrained 8 million-parameter Transformer (encoders only).

Table 3 .

 3 4: BioBERT performance on MD, NER (semantic types) and EL on Med-Mentions, with State Of The Art (SOTA) entity linker[START_REF] Bhowmik | Fast and effective biomedical entity linking using a dual encoder[END_REF] and TaggerOne(Leaman and Lu, 2016, referenced in the original MedMentions paper byMohan and Li) as baselines for comparison.

	Task	Precision Recall Micro-F 1
	SOTA Entity Linking	0.53	0.54	0.53
	TaggerOne	0.47	0.44	0.45
	Mention Detection	0.66	0.71	0.68
	Semantic Type NER	0.60	0.63	0.60
	Entity Linking	0.22	0.21	0.21

Table 4 .

 4 .1, this lack of re-training data has prevented the language model from successfully integrating the KAR, with a masked LM perplexity several orders of magnitude larger than BERT BASE , BERT LARGE , and the KnowBert models produced by Peters et al. 1: Masked Language Model perplexity for both BERT models, the KnowBert variants produced by[START_REF] Peters | Knowledge enhanced contextual word representations[END_REF], and KnowBert-UMLS.

	Model	Perplexity
	BERT BASE	5.5
	BERT LARGE	4.5
	KnowBert-Wiki	4.3
	KnowBert-Wordnet	4.1
	KnowBert-W+W	3.5
	KnowBert-UMLS	10387.7

Table 4 .

 4 .2, we compare our performance versus four BERT-based 2: Performance of BERT-based language models on the n2c2 NER task, measured as Micro-averaged strict Precision, Recall and F1. Results for BioBERT, clinicalBERT and BlueBert from[START_REF] Fraser | Extracting umls concepts from medical text using general and domain-specific deep learning models[END_REF].

			Model		P	R	F1
			BERT BASE	0.85 0.87 0.86
			BioBERT		0.86 0.88 0.87
			clinicalBERT	0.87 0.88 0.88
			BlueBERT	0.88 0.90 0.89
			KnowBert-UMLS 0.80 0.81 0.80
	Sequence	total	abdominal hysterectomy and	bilateral	salpingo-oophorectomy	.
	True	B-treatment I-treatment	I-treatment	O	B-treatment	I-treatment	O
	Predicted B-treatment I-treatment	I-treatment	O	B-treatment	I-treatment	O

Table 4

 4

.3: Example of correct NER predictions by KnowBert-UMLS pulled from the n2c2 evaluation set. models, namely BioBERT

[START_REF] Lee | BioBERT: a pretrained biomedical language representation model for biomedical text mining[END_REF]

, clinicalBERT

[START_REF] Alsentzer | Publicly available clinical BERT embeddings[END_REF]

, BlueBERT

[START_REF] Peng | Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets[END_REF]

and BERT BASE , all fully fine-tuned on the NER task with the same linear classifier architecture. The performance of our various baselines were taken from

[START_REF] Fraser | Extracting umls concepts from medical text using general and domain-specific deep learning models[END_REF]

.

Table 4 .

 4 al. (2019) in table 4.5.

	Sequence Estimated	blood	loss	was 100 cc
	True	B-problem I-problem I-problem	O	O	O
	Predicted	O	B-problem I-problem	O	O	O

4: Example of incorrect NER prediction by KnowBert-UMLS pulled from the n2c2 evaluation set.

Table 4 .

 4 1 gigawords of 6: Pre-Training Corpus Size (Billions of Words) by Type for Baselines Versus KnowBert-UMLS.

	Model	Biomedical General Knowledge integration
	BERT BASE	0.0	3.1	No
	PubMedBERT	3.2	0.0	No
	BlueBERT	4.5	3.1	No
	BioBERT	18.0	3.1	No
	UmlsBERT	18.5	3.1	Yes
	KnowBert-UMLS	2.2	3.1	Yes

Table 4 .

 4 11: Performance on the modified CoLA task, Matthews' Correlation and Micro-F 1 .

	1

Table 5 .

 5 2: Table comparing results of models incorporating synthetic text in their training corpus versus those of models trained exclusively on natural language, on linguistic acceptability (CoLA) and inference (SNLI) tasks. Results are averages over 4 experiments. The underlined result is the best for each task.

	Model	CoLA (M. Corr.) SNLI (F 1 )
	BERT BASE	63.17	90.58
	BERT PMC	64.11	90.38
	BERT SCGU	61.89	90.44
	BERT 50% Hybr	61.62	90.28
	BERT 100% Hybr	63.86	90.20

Table B .

 B 1: Hyperparameter values for the various tasks.

			Rate Weight Decay Batch size Epochs
	Pre-training	2e-6	0.01	4096	1
	ChemProt	2e-5	0.01	24	30
	PubMedQA	2e-5	0.01	32	5
	CoLA	2e-5	0.01	32	10
	SNLI	2e-5	0.01	32	5

111 112

FLOP, FLOPs, or floating-point operations, are a measure of amount of computation for a task, which is independent of time and hardware. It is not to be confused with FLOPS, FLOP/s, or floating-point operations per second, which are a measure of computation speed of a piece of computer hardware.

https://lambdalabs.com/blog/demystifying-gpt-3

https://neoteric.eu/blog/how-much-does-it-cost-to-use-gpt-modelsgpt-3-pricing-explained/

https://www.tensorflow.org/datasets/catalog/c4 https://github.com/allenai/allennlp/discussions/5056

For a trivial example (albeit of limited usefulness), it could encode the first token t 0 in the sequence into its representation h 0 and set h t+1 = h t ∀t ≥ 0. More formally, h t+1 = σ(W h h t + W x x t+1 + b h ) where h 0 = x 0 , W h = I the identity matrix, and W x = O the 0s matrix.

See equation 2.1 for definition of c t

seeSection 2.3

There are 127 semantic types in UMLS. UmlsBERT uses 44, but it is not made explicit which semantic types were chosen or why.

Some definitions omit the MD sub-task. However, there is to my knowledge no better term for the coupling of both Mention Detection and Entity Disambiguation. We do not go over disambiguation-only approaches as their actual EL performance is heavily reliant on the accuracy of the upstream MD step.

This annotation scheme assumes entity mentions are continuous. Handling discontinuous entities requires complexification of the annotation scheme which can be counterproductive. This is a topic of ongoing research(Dirkson et al.

, 2021).

There approximately 6.7 million Wikipedia articles versus 3.3 million concepts and 15 million concept names in UMLS as of the writing of this dissertation

Perplexity is computed as the exponential of the cross-entropy loss, and is a standard measure of how well the language model predicts samples.

In addition to the results presented here, we have evaluated our models on the i2b2

2010/n2c2[START_REF] Uzuner | i2b2/va challenge on concepts, assertions, and relations in clinical text[END_REF] NER task, but do not report results as the performance differences were not statistically significant.2 specifically, the 0.8.2-fp16_e_s3 version, which has been deprecated since January 2021

The commonly used term in the literature is Large Language Model (LLM, omitting the generative aspect). There is, however, debate on whether non-generative models should be included in the category. We introduce the more-specific term LGLM to settle the debate and avoid ambiguity.

"Non-conflicting" is a lesser constraint than "identical" as preferences are not strictly ordered.

ChatGPT is "just" a transformer-decoder based auto-regressive language model fine-tuned with Reinforcement Learning from Human Feedback (RLHF) with a web interface.

https://edition.cnn.com/2023/01/25/tech/cnet-ai-tool-news-stories/ index.html

https://www.businessinsider.com/ai-chatgpt-jobs-replaced-by-techtranslator-2023-9

6 https://edition.cnn.com/2023/05/04/tech/writers-strike-ai/index.

html 7 https://time.com/6256529/bing-openai-chatgpt-danger-

alignment/ 8 https://www.brusselstimes.com/430098/belgian-man-commits-suicidechatgpt

One is generally more likely to achieve one's current goals if one remains operational, one's objectives do not change, and one has more resources such as computing power.

Acknowledgements

Biomedical Relation Extraction

This is a sequence classification task, wherein two entities per sequence are marked with special characters, and the model must determine which of five relation types (or no relation) exists between them. Scores for BioBERT, PubMedBERT and BlueBERT are self-reported scores of the overall best-performing version of each model. The performance of BERT BASE was measured by us using the version of the ChemProt corpus distributed by [START_REF] Peng | Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets[END_REF].

Due to the entities being marked in all versions of the training set used by our baselines, the importance of a good model for grammar is lessened with respect to other tasks. We therefore do not expect general language understanding to be highly predictive of performance on this task. Rather, knowledge of biomedical entities and their relations is expected to be of greater importance. We therefore expect KnowBert-UMLS, which seeks to acquire specifically this type of knowledge, as well as the heavily pretrained BioBERT model, to perform well on this task, with BERT BASE performing worst. Our results in Table 4.8 largely correspond to what we expected. PubMedBERT, however, which was pretrained only on biomedical text, performs better than expected, implying out-of-domain pre-training may in fact be detrimental.

Once more, we observe that KnowBert-UMLS outperforms BERT, implying that specialization was successful, yet it underperforms in comparison to other pretrained models. BioBERT being outperformed by PubMedBERT is unexpected, but may be due to the small pre-processing differences in the version of the corpus used by BioBERT.

In order to capture this ambiguity, in addition to the reference label, each sentence has been manually labeled by five human reviewers. In this case, the label is 'entailment', but two out of five reviewers labeled the relationship between these sentences as 'neutral'. If we accept the reviewers' answers as valid predictions, the micro-F 1 scores of KnowBert-UMLS and BERT (that is, the specific instances used in this analysis) on this task are 96.80 and 96.88 respectively, meaning only approximately one in four errors made by KnowBert-UMLS and BERT were in disagreement with all reviewers.

We decide to examine the mistakes made by both models and attempt to identify patterns. No clear tendencies could be found regarding the incorrect label predictions, but analyzing the instances themselves, we could group mistakes into four major types:

• Blunders, for which no information other than what is stated in the text is required to make a decision.

• Common Sense (CS) mistakes, which require some reasoning and/or a nontrivial piece of real-world knowledge.

• Technically Correct (TC) predictions, in which the model's answer could be considered correct based on an arguably valid interpretation of the text.

• Not-an-Error (NaE), for which we agree with the model and disagree with the label, or the input text contains a major corruption.

We provide examples of the aforementioned categories in Appendix A.1.1. After manual examination of the models' failure cases, we report a breakdown of mistakes by type in Table 4.10. The main recurring pattern seems to be for KnowBert-UMLS to lack realworld knowledge but stick to more straightforward interpretations of sentences. Illustrative examples for CS and TC mistakes are given in appendices A. 1.2 and A.1.3 respectively. This may indicate that KnowBert-UMLS suffers from some amount of catastrophic forgetting on real-world knowledge, and perhaps is less prone to noticing and fixating on details which could skew its understanding of the text.

Model

A.1.3 . Technically correct answers by BERT on the SNLI task

Pre. A boy dressed in a plaid kilt with a brown hat wields a long pole.

Hyp. The boy is holding a samurai sword.

True: contradiction | Predicted: neutral One could wield both a long pole and a samurai sword, but this is an unlikely scenario.

Pre. 4 children and 1 adult look at an armadillo on a grassy hill with 2 trees.

Hyp. A family visits the zoo.

True: neutral | Predicted: contradiction Both sentences are not mutually exclusive, but in fairness to BERT, zoos rarely have animal pens on grassy hills.

Pre. A roofer in a gray sweatshirt and orange hat walks on a unfinished roof at a lake-side home.

Hyp. The roofer is putting on shingles.

True: neutral | Predicted: contradiction Shingling and walking are mutually exclusive actions at any given time, but the action in the hypothesis should be interpreted as a continuous process.

KnowBert-UMLS makes correct predictions for these instances.

A.2 . BERT and KnowBert-UMLS predictions on the CoLA task

This section lists all of the false positive and false negative predictions by BERT and KnowBert-UMLS on the CoLA task. These predictions are separated into six itemized lists depending on whether they are false positive or false negative predictions, and whether they were made by BERT, KnowBert-UMLS, or both.

False negatives which use '→' as a bullet, despite being technically correctly labeled when considering all regional variations of spoken English, are considered mislabeled and are re-labeled as true negatives for the purpose of estimating model performance in a literary setting (See section 4.3.2.4).

Shared false positives

• The more you would want, the less you would eat.

• The more does Bill smoke, the more Susan hates him.

• Mickey looked up it.