Information extraction (IE) is defined as the identification and extraction of elements of interest, such as named entities, their relationships, and their roles in events. For example, a web-crawler might collect open-source documents, which are then processed by an IE treatment chain to produce a summary of the information contained in them. We model such an IE document treatment chain as a Markov Decision Process, and use reinforcement learning to allow the agent to learn to construct custom-made chains "on the fly", and to continuously improve them. We build a platform, BIMBO (Benefiting from Intelligent and Measurable Behaviour Optimisation) which enables us to measure the impact on the learning of various models, algorithms, parameters, etc. We apply this in an industrial setting, specifically to a document treatment chain which extracts events from massive volumes of web pages and other open-source documents. Our emphasis is on minimising the burden of the human analysts, from whom the agent learns to improve guided by their feedback on the events extracted. For this, we investigate different types of feedback, from numerical rewards, which requires a lot of user effort and tuning, to partially and even fully qualitative feedback, which is much more intuitive, and demands little to no user intervention. We carry out experiments, first with numerical rewards, then demonstrate that intuitive feedback still allows the agent to learn effectively. Motivated by the need to rapidly propagate the rewards learnt at the final states back to the initial ones, even on exploration, we propose Dora: an improved version of one of the most standard of reinforcement learning algorithms, Q-Learning. Finally, we apply BIMBO and Dora to a different setting -that of the detection of objects in an image.

Foreword

This thesis was carried out under the French ANRT (Association Nationale de la Recherche et de la Technologie or National Association for Research and Technology) system, specifically the CIFRE (Conventions Industrielles de Formation par la REcherche or Industrial Agreements for Training through Research). The CIFRE 's objectives are to develop partnerships between public research laboratories and businesses, and to help doctors to find employment outside academia.

The proportion of time that the student should spend working on the thesis as opposed to working for the company is not specified by the ANRT, but in my case, I was able to spend around one half to two thirds of my time on my thesis or related project work.

During the three years of this thesis, I officially changed company several times. At first it was just name changes from Cassidian (originally EADS Defence and Security) to Airbus Defence and Space. Then a major change occurred during the second year of my thesis, when Airbus Defence and Space were faced with the option of either closing or selling the site on which I worked. A buyer, Cordon Electronics, was eventually found, and the transfer finalised in October 2015 when we became Cordon Electronics DS2i.

This transfer posed a technical challenge as the in-house WebLab services such as translation, and in-depth document analysis (see chapter 4) that we originally planned to introduce into the chain belong to Airbus Defence and Space. They kindly granted me special authorisation to keep the service encapsulating GATE and the WOOKIE ontology developed by [START_REF] Serrano | Vers une capitalisation des connaissances orientée utilisateur: extraction et structuration automatiques de l'information issue de sources ouvertes[END_REF] that I had started using for my tests. All forward in time to update the current state. . . . . . . . . . . 12.2 Sutton and Barto (1998)'s backward (mechanistic) view: looking backward in time to update the previous states. . . . . . . 12.3 A frustrating exploration gap in a very long chain, just before a large reward, where standard QL(λ) cuts the trace, preventing the back-propagation of the information. . . . . . . . . . . 12.4 An example of how QL(λ) cuts the trace on an explore, preventing the back-propagation of useful learnt values. . . . . . 12.5 An example of how Dora keeps the trace intact on an explore which turns out with hindsight to be a "best action", thus allowing the back-propagation of useful learnt values. . . . . . 12.6 Dora and QL(λ) were run on the same randomly generated MDP of 80 states and 6 actions. Results were measured using the episodic distance averaged over 50 runs. . . . . . . . . . . 12.7 "Long thin" MDP. . . . . . . . . . . . . . . . . . . . . . . . . 12.8 Dora and QL(λ) were run on the same Long Thin MDP (Figure 12.7) of 15 states and 2 actions. Results were measured using the episodic distance averaged over 50 runs. . . . . . . .

Introduction

Imagine a car factory that produces cars with one wheel on the roof, one sticking out at the side, and two on the ground. The customers of this rather odd factory have to change the wheels every time they buy a car. It sounds improbable, but this can be the case with Information Extraction (IE) systems. Vast amounts of data are trawled, and the system tries to provide a summary of the useful information. Any flaws in the extraction of this information mean that the summary is inaccurate, and the end user will need to correct it. The factory realises that the customers are not happy, and so they provide an expert who can produce custom cars. The customer just needs to take the time to come into the factory and explain exactly what he wants. The IE system (often a chain of treatments) can likewise be configured by an expert in discussion with the user (hoping that the user doesn't change his mind once he sees the results).

The factory decides to increase user satisfaction by providing tools that allow the customer to change the factory layout themselves, and to give the customers training in car-assembly. The user of the IE system has been given special languages to interact with the IE chain, but he can lack the expertise to appreciate the impact of his choices.

The factory then takes a radical decision. It creates a new customer feedback department. They issue opinion polls, asking users to mark the features of their cars from 1 to 1000, or to rank them in order of preference. They then use these questionnaires to build tailor-made cars for each customer. We can see that obtaining feedback from the users could help to improve the IE services, but is it really natural to ask them to provide it explicitly and numerically?

Finally, the factory installs a monitoring device inside the car, which automatically tells the feedback department exactly how the car is used, how it is modified, which colour the customer paints it on which day, which features make the customer smile most, etc. At last, they can produce tailormade cars for the customer, without having to interrogate them, train them, or get them to fill in lengthy questionnaires. 1 2 CHAPTER 1. INTRODUCTION Our objective is to install a "monitoring device" into an IE chain. It will observe the users actions, and requires only intuitive natural feedback to improve the chain (although, of course, it can be given numerical feedback too).

Context

Many people rely on IE systems, including Open Source INTelligence (OS-INT) analysts. In the past, the OSINT analyst's task was finding hidden information. Now, faced with ever-increasing volumes of web pages and other open-source multimedia, the challenge is to find pertinent information from the huge wealth of data available.

Many specialised treatment chains have been developed to ease this task, but in the particular platform that we examine 1 , open-source multimedia are inserted continuously into a series of web services which aim to extract events (e.g., terrorist attacks) and their characteristics (date, place, agents, etc.) for OSINT assessment and surveillance.

The chain, defined by experts, consists of a series of individual treatments, such as the detection of the multimedia format, transcription, sentiment analysis, language recognition, translation, extraction of named entities and events using nouns and verbs as triggers, etc. An example of a possible multimedia treatment chain is shown in Figure 1.1, and we invite you to see section 4.1 for more details of the document treatment chain that we used for our tests.

It is clear that the output of this chain, the named entity and event extractions, cannot be perfect. Firstly, because of the huge diversity of multimedia on the Web, there cannot be a single optimal chain for all of them. Secondly, imagine, for example, a school blog recounting "the bombardment of a gold target by ions during an atomic physics demonstration at the school science fair". The treatment chain could erroneously recognise and extract an atomic bomb attack. Thirdly, the need to treat multimedia coming from all over the world in diverse languages entails the use of dictionaries and translation services whose quality is variable. Finally, an error early in the chain can be amplified as it goes further down the chain, for instance an error in the transcription of a voice recording will probably cause errors in the translation of that transcription, etc. The analysts are therefore typically forced to correct a posteriori the events extracted. These corrections are 1 As part of the industrial challenge underlying the CIFRE contract. In parallel, the languages spoken are identified and the voice recording is transcribed. The native language extractor is used to summarise the information contained in the transcript, whilst the transcription is also translated. The polarity of the opinions expressed, and their subjectivity is assessed, and an English extractor used. In all cases the information is indexed in a database.

CHAPTER 1. INTRODUCTION

There is currently no way to improve the chain over time without the intervention of an expert in consultation with the analysts.

So why don't we let the analyst reconfigure the chain themselves? First and foremost, they simply do not have the time to make the alterations. Even if they did, they are not system experts, and can lack the expertise to appreciate the impact of their choices. Nor do they have an in depth knowledge of the services to correctly determine which is best to use in a given situation.

Our objective is to remedy this situation by providing a "feedback department" for the chain which receives its feedback as the analysts consult and correct the system output. It is responsible for modifying the behaviour of the chain in real time, ensuring its continuous improvement, and reducing human effort by decreasing the error rate. For example, if the analyst corrects an event, we could give an explicit numeric feedback based on the distance between the event as corrected and that which was extracted. However, this numerical feedback is neither natural nor easy for the analyst to define. It is not intuitive to determine precisely that "Correcting five letters in a partially correct name is worth 0.71, but deleting an incorrectly extracted place is worth 0.69".

We therefore want to advance to improving the chain based on qualitative, intuitive feedback by "monitoring" the user actions, applying naturally expressed user preferences which can be gathered or changed very easily, such as "I prefer a partial, or even a false summary to seeing nothing extracted" and "I prefer the treatment to be as fast as possible", and applying them to the extraction results.

Contribution

We formalise the improvement of a document treatment chain which extracts events from web pages and other open-source documents as a reinforcement learning (RL) problem (chapter 5), and the treatment chain itself as a Markov Decision Process (MDP) (chapter 6).

We build a platform, BIMBO (Benefiting from Intelligent and Measurable Behaviour Optimisation), which can modify the treatment chain in real time, based on the end user feedback (chapter 7).

BIMBO is built in a modular fashion, which means that we easily test different RL algorithms and reward mechanisms, and measure their impact on the learning.

We investigate three different types of user feedback:

ORGANISATION OF MANUSCRIPT

• In the first, the agent receives numerical rewards on the quality of the treatment (chapter 8). These experiments show that automatically learning how to chain and parametrise services is perfectly feasible. The problem is that numerical rewards are very difficult for the human user to calibrate.

• In the next two, we use qualitative feedback, for which we present a novel algorithm in chapter 9.

-We first use a partially qualitative formalisation (chapter 10, subsection 10.1.1). The agent is rewarded with a semi-qualitative feedback which gives an indication of the strength of the user preferences, rather than a specific numeric value. This proves very effective, but still requires some calibration.

-Then in chapter 10, subsection 10.1.2 we use a fully qualitative and intuitive user feedback. This is less informative for the agent, but requires no calibration and can still give very good results.

You will see in chapter 8 that during the learning process, the AI may try several hundreds of actions during one treatment. This leads to a very long chain of state / action pairs (see chapter 5), or "paths" before it finally receives a reward. Frustrated by the length of time that this reward was taking to back-propagate along these very long paths, we suggest an improvement to the algorithm Q-Learning, Dora, which loses less information when exploring (chapter 12).

We initially apply BIMBO to a document treatment chain, but we demonstrate the flexibility of our approach by applying BIMBO and Dora to the parametrisation of image analysis services (chapter 13). In fact, BIMBO is not dependent on the web services used, nor the type of input, and can therefore equally be used for the study of any process where we can define the states in terms of the system and the "thing" being processed, and where the actions can influence the parameters and / or the process itself.

Organisation of manuscript

We start by giving an overview in chapter 2 of work which is related to our own. This chapter has been kept intentionally relatively short, as we interleave the references with the rest of the manuscript, going into detail of particular examples in the relevant sections.

CHAPTER 1. INTRODUCTION

In the main part of this manuscript, Part I, we examine the industrial challenge which motivated this work (chapter 4) and our response to it as a reinforcement learning problem (chapter 5), modelled as a Markov Decision Process (MDP) (chapter 6). In chapter 7, we outline the platform that we constructed, and the general experimentation framework. We detail the first experiments that we carried out with numeric, potentially sporadic, simulated feedback data (chapter 8), which show that automatically learning how to chain and parametrise services is perfectly feasible. We recognise that users do not naturally provide numeric feedback, and so in chapter 9 we present our contributions towards a novel algorithm which combines qualitative feedback with reinforcement learning, the design of which was led by Hugo Gilbert (LIP6, Université Pierre et Marie Curie, Paris) with our collaboration. We then present our experiments with two types of intuitive feedback (chapter 10), which we show are perfectly feasible solutions, while requiring little to no tuning.

In Part II, we detail two spin-offs of our research. The first (chapter 12) is Dora, an improvement to the reinforcement learning algorithm Q-Learning, which loses less information when exploring. The second is a demonstration of the flexibility of our approach, applying BIMBO and Dora to the detection of objects in images (chapter 13).

Finally, in Part III: Conclusion we lower the curtain with some thoughts

Chapter 2

Related work

Introduction

Throughout this document, where relevant, we examine the existing academic work and draw parallels with our own. Nevertheless, in this chapter, we give a short, self-contained summary of work related to ours.

Even though the applications of our work are broader, as we said in chapter 1: Introduction, this thesis was initially motivated by the need to extract events from open-source documents for the Open Source INTelligence (OSINT) community. In section 2.2, therefore, we examine Information Extraction (IE), specifically the extraction of events.

To our knowledge, an agent which constantly learns and self-improves from natural human feedback to dynamically construct and parametrise a chain of services has not yet received academic attention. Nevertheless, several of the base elements have been studied. We summarise some of these elements here, and enter into more detail in the following chapters.

We are all familiar with and use daily the results of treatment chains, consisting of an input, a process, and a result. A production chain in a factory, for example might have pieces of metal and plastic as input, programmed robots to execute the process, and finally produce a car. Chaining together services to attain a desired result is therefore not a new concept, and we examine several approaches in section 2.3.

Neither, evidently, are we the first to model a process as a Markov Decision Process (MDP); section 2.4 gives examples of some approaches to modelling a problem using MDPs: the policy can be defined in advance by an expert, it can be learnt from existing data, or it can be learnt "on the job" from real-time observations. Finally, using human feedback, both explicit and implicit for model-based learning has also been explored. Our objective is to enable the human users to give an intuitive, natural feedback instead of a numerical reward. In section 2.5, we examine some of the work that has already been carried out in this domain. 10 CHAPTER 2. RELATED WORK

Information Extraction

So we examine Information Extraction (IE), but what is it, and how does it differ from Information Retrieval (IR) and Information Acquisition (IA) (which we consider out of the scope of this thesis)? [START_REF] Cowie | Information extraction[END_REF] compare IR to a combine harvester which trawls vast fields to harvest potentially useful raw material, and IE to the refining process which transforms it. Continuing this culinary analogy, we could say that IA is like mushroom picking in the forest -you are searching for something specific. A web-crawler, for example, could be an IR engine collecting open-source documents, which are then processed by an IE treatment chain which produces a summary of the information contained in them. An IA engine then might search for further or missing information (which could be derived from several other data) on one of the elements of interest.

More specifically, [START_REF] Hobbs | Information extraction[END_REF] define IE as the identification and extraction of elements of interest, such as the named entities, their relationships, and their roles in events. Event extraction from text, the "who did what to whom, how, when, where, why and with what?", is the subject of numerous studies. For instance, [START_REF] Hogenboom | An Overview of Event Extraction from Text[END_REF] give an overview of three different extraction methods: data-driven, expert knowledge-driven and hybrid, and [START_REF] Serrano | Combinaison d'approches pour l'extraction automatique d'événements[END_REF] carry out a comparative study of statistical and linguistic rules for automatic event extraction.

These IE chains are expensive to build and maintain, and components are often not reusable. For instance, [START_REF] Cowie | Information extraction[END_REF] estimated that the cost of transferring an existing system to a new domain would take from three to six person-months. This transfer is long, and if it has to be carried out by an expert, would be very costly. This led them to state that domain-specific IE systems were only acceptable if they did not require a specialist to tune them. [START_REF] Chiticariu | Rule-based information extraction is dead! long live rule-based information extraction systems![END_REF] state that some of the main costs of developing an IE system are the labour involved in understanding and defining the business problem, and writing the extraction rules. IE platforms, therefore should ideally be multi-lingual [START_REF] Lejeune | Veille épidémiologique multilingue: une approche parcimonieuse au grain caractere fondée sur le genre textuel[END_REF] and multi-media [START_REF] Ji | Challenges from information extraction to information fusion[END_REF], with standard rule definition [START_REF] Chiticariu | Rule-based information extraction is dead! long live rule-based information extraction systems![END_REF] and should represent the knowledge acquired on several levels [START_REF] Lee | Enhancing multi-lingual information extraction via cross-media inference and fusion[END_REF].

Not only is the chain expensive to build, it is also difficult to test. The difficulty is that the annotated corpora that currently exist (e.g. ESTER2, Quaero, etc.) tend to be domain-specific, or not broad enough, covering only named entities [START_REF] Fort | Les ressources annotées, un enjeu pour l'analyse de contenu: vers une méthodologie de l'annotation manuelle de corpus[END_REF], and not those elusive relations between the actors and the events [START_REF] Ludovic | Approches supervisées et faiblement supervisées pour l'extraction d'événements complexes et le peuplement de bases de connaissances[END_REF]. Annotating a document by hand is

CHAIN CONFIGURATION

time-consuming and error-prone, and the automatic annotation of a corpus for IE is complex [START_REF] Riloff | Automatically generating extraction patterns from untagged text[END_REF]. The Automatic Content Extraction (ACE) Program was a competition (Linguistic Data Corporation (LDC), 2016;[START_REF] Doddington | The automatic content extraction (ACE) programtasks, data, and evaluation[END_REF] which aimed "to develop extraction technology to support automatic processing of source language data . . . " Intitially, the ACE research objectives were the detection and characterization of named entities, but from phase 3 onwards, ACE sought to pit IE systems against each other to correctly automatically annotate the events and named entities in documents [START_REF] Ji | Refining Event Extraction through Cross-Document Inference[END_REF].

The particular IE chain that we use extracts events for open-source intelligence. [START_REF] Serrano | Events extraction and aggregation for open source intelligence: from text to knowledge[END_REF] explain the motivation behind the development of these extractions, and propose a data-fusion method to combine two events based on mutual completion of information and their proximity in an ontology. However, there is no consensus as to how to define an event, nor how to represent the entities and events so that they are really open-domain and able to be co-referenced [START_REF] Bejan | Unsupervised event coreference resolution[END_REF][START_REF] Serrano | Vers une capitalisation des connaissances orientée utilisateur: extraction et structuration automatiques de l'information issue de sources ouvertes[END_REF][START_REF] Byrne | Populating the semantic web: combining text and relational databases as RDF graphs[END_REF][START_REF] Dutkiewicz | Knowledge-based highly-specialized terrorist event extraction[END_REF][START_REF] Van Hage | Design and use of the Simple Event Model (SEM)[END_REF], nor even on how to measure the similarity between named entities, events, and their relations [START_REF] Saval | Temporal pattern, spatial and semantic for the discovery of relationships between events[END_REF][START_REF] Moreau | Robust similarity measures for named entities matching[END_REF][START_REF] Wang | Efficient approximate entity extraction with edit distance constraints[END_REF][START_REF] Cohen | A comparison of string metrics for matching names and records[END_REF][START_REF] Wang | Entity matching: How similar is similar[END_REF][START_REF] Cohen | Very Fast Similarity Queries on Semi-Structured Data from the Web[END_REF][START_REF] Kumaran | Using names and topics for new event detection[END_REF][START_REF] Lao | Reading the web with learned syntactic-semantic inference rules[END_REF][START_REF] Lao | Relational retrieval using a combination of path-constrained random walks[END_REF][START_REF] Bellenger | An Evidential Approach for Modeling and Reasoning on Uncertainty in Semantic Applications[END_REF][START_REF] Gardner | Incorporating vector space similarity in random walk inference over knowledge bases[END_REF]. [START_REF] Steinberger | An introduction to the europe media monitor family of applications[END_REF] have made a step in this direction with The Europe Media Monitor platform, extracting events from world-wide news feeds, enabling them to be co-referenced, viewed on a map, translated, etc. Similarly New Event Detection (NED) aims to monitor a stream of news and trigger alerts when a new topic or event is detected. This problem centres around the comparison of incoming information sources with an existing knowledge base of all those seen in the past in order to detect new topics. NED techniques (see e.g. [START_REF] Kumaran | Text classification and named entities for new event detection[END_REF] for an overview) are often based on the similarity between terms used in the documents, or their temporal difference (how much older one story is than the other).

Chain configuration

Given that these IE treatment chains are so costly to build, several attempts have been made to do it automatically.

The chain can be planned and fixed in advance, as in [START_REF] Saïs | Proceedings of SOS-DLWD 2013[END_REF] who model service composition using AI-Planning and Petri Nets for an IA system. The service compositions that they compute help the user to fill in CHAPTER 2. RELATED WORK forms. The inputs and outputs of each e-service are known in advance, and it is a planning task to string them together.

The chain can even adapt itself depending on the input, for example, the physical characteristics of documents can be successfully used to construct adaptive, modular chains for Xerox photocopiers in real time [START_REF] Fromherz | Model-based computing for design and control of reconfigurable systems[END_REF]. As with the service composition task above, each module, or operation, within the photocopier is defined in terms of its possible inputs and the time taken to produce each possible output. A system controller is given the task information or document specification, for example "five collated, stapled, double-sided copies of a 10-page document". From this specification, the controller can plan the optimal sequence of operations which produce this result, and then schedule them. The scheduling consists, for example, of leaving a gap between sheets where necessary to avoid jams because one operation takes longer than the others, or of interleaving operations where possible to save time.

The chain can also be modified "on the fly" by an expert. For instance, [START_REF] Doucy | Méthodologie pour l'orchestration sémantique de services dans le domaine de la fouille de documents multimédia[END_REF] use the platform WebLab to construct a modifiable document treatment chain. The services can be chosen from a central directory and substituted as required using BPEL (Business Process Execution Language) to change the underlying XML, and the WSDL (Web Service Definition Language) of each service to ensure the substituted service is compatible. Similarly [START_REF] Rodrigues | Towards an engine for coordination-based architectural reconfigurations[END_REF] allow the reconfiguration of a chain of webservices through a domain specific language, ReCooPLa (developed as an Eclipse plugin). This allowed them to formalise the graph of potential interactions between the services in a Service Oriented Architecture (SOA). The software architect can then use this language through a graphical interface to try out different configurations, whilst ensuring that the whole remains coherent.

The particular chain that motivated our work is an IE document treatment chain. Open-source documents (the input) are fed into a software chain of web services which add information (or not) to the document (the process), and finally that information is presented in a user-friendly format (the output). For an overview of similar processing chains that are provided as web services, we invite you to consult [START_REF] Bibliography Ogrodniczuk | Linguistic Processing Chains as Web Services: Initial Linguistic Considerations[END_REF]. The inputs, duration, and outputs of each web service are unknown in advance, and so the agent must learn to build the chain step by step, according to the information currently available.

MODELLING 13

Modelling

Any uncertain environment where decisions have to be taken sequentially can potentially be modelled as a Markov Decision Process (MDP).

For instance, in the financial world, [START_REF] So | Modelling the profitability of credit cards by markov decision processes[END_REF] model the credit limit of clients for banks, enabling them to estimate the profit that they might gain by increasing the credit limit. The states reflect the client's current behaviour score and their current credit limit. The actions available are to increase the credit limit by various amounts. The reward received by the agent is the profit made for the bank.

In a medical environment, a course of treatment is influenced by the patients current state of health, their response to the treatment, and the anticipation of future treatment that may be necessary. [START_REF] Schaefer | Modeling medical treatment using markov decision processes[END_REF] detail some of the applications which have already modelled medical treatments as MDPs. Staying in the medical domain, [START_REF] Ma | Modelling with PRISM of intelligent system[END_REF] examine a hand-washing process for dementia patients using action policies which are pre-defined by an expert. Observations are provided by sensors, e.g. tracking user's hands, towel and soap, and the actions are the prompts given to the patient. The patient characteristics, like the action policy, are specified by an expert. They use PRISM (a stochastic model checking tool) to compare the efficiency of the policies and to identify factors that play important roles. This allows them to establish general rules for choosing policies for different users.

Closer to home, in the field of information acquisition (IA), [START_REF] Kanani | Resource-bounded Information Acquisition and Learning[END_REF] tries to minimise the cost of filling a partially complete database where recuperating the required information is expensive. She models this Resourcebounded Information Extraction (RBIE) problem as an MDP. The probabilistic model for extracting the values from the external sources, and the relative importance of each piece of information is known in advance. The state of the database at each timestep, plus the intermediate results gained so far defines the states of the MDP, and the actions are query, download and extract. The order of these actions is fixed, and the structure of the queries defined in advance, but the queries are filled in dynamically based on the information sought. In constrast our agent must learn the probabilistic model, and the optimal order in which to perform the actions, which may vary from document to document.

CHAPTER 2. RELATED WORK

Learning from users

Humans share their knowledge with each other in many ways: through speech, writing, demonstrations, etc. They can also transfer this knowledge to machines by training them, either to replicate the skills of the human, or to learn their preferences to aid them in performing a task. [START_REF] Bratko | Learning qualitative models[END_REF] argue that replicating human skills through a dialogue is not a cognitively easy task. Often the human is incapable of describing precisely what he does numerically, but will be more at ease expressing himself qualitatively. They only require qualitative observations of the changes in the environment, formalised as Qualitatively Constrained Functions (QCFs) where: Z = M +,-(X, Y ) means that Z monotonically increases in X and decreases in Y . It can be difficult, however, to tell how Z will change. For example, if both X and Y increase, then Z can increase, decrease, or stay unchanged. There must also be a large enough delta for the difference to be detectable [START_REF] Saaty | Relative measurement and its generalization in decision making why pairwise comparisons are central in mathematics for the measurement of intangible factors the analytic hierarchy/network process[END_REF]. [START_REF] Bratko | Learning qualitative models[END_REF] also point out that the actions of the human are often a better indicator of how to perform the task, and "cloning" the human by examining these traces enables a knowledge transfer, not only to the machine, but to other humans. These traces can also be used to evaluate user profiles. For example, [START_REF] Karami | Apprentissage de connaissances d'adaptation à partir des feedbacks des utilisateurs[END_REF] use two algorithms for this. The first updates itself using actual user-interactions or traces. The second is faster, but with a higher risk of error, as it tries to generalise existing rules to new interaction contexts and / or user profiles. It determines the relevant attributes of the state by applying previous knowledge about the rewards gained by performing a given action, and the values of the attributes of the state at that time.

Machines can learn from qualitative feedback in a whole range of ways, from staying a silent observer to being an active participant, asking questions and trying to carry out the task itself.

Some examples of silent observers can be seen in the improvement to translation models. Users correct or evaluate the output, and the translation model is updated as a result [START_REF] Formiga | Leveraging online user feedback to improve statistical machine translation[END_REF][START_REF] Snover | A study of translation edit rate with targeted human annotation[END_REF]. [START_REF] Culotta | Corrective feedback and persistent learning for information extraction[END_REF] introduce us to the idea of using the corrections to make a permanent difference to the system, referring to the corrections or evaluations as "corrective feedback", and the updating of the model as "persistent learning". Some systems allow the user to interact more directly with the system through a specialised language [START_REF] Chai | Efficiently incorporating user feedback into information extraction and integration programs[END_REF] to allow users to give feedback by correcting the extracted data, or to change the parameters of the extraction system. Changes are then propagated through the database and the next extraction crawls.

The agent can also learn by observing the trainer's actions on the same tasks. This allows it to extract a reward function simply by observing optimal behaviour. This is known as inverse reinforcement learning (IRL) and [START_REF] Ng | Algorithms for inverse reinforcement learning[END_REF] present several algorithms for recovering reward functions from the observed behaviour. The premise is that the users have already defined a reward function whether consciously or not, and the agent has to try to guess it. [START_REF] Loftin | Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning[END_REF] take this idea further by allowing the agent to learn, not only from explicit user feedback where the user specifies a reward, but also by deducing the implicit or withheld feedback, where the user does not provide a reward, but where one could be inferred. Their users are non-technical, and will not necessarily provide feedback in a homogeneous manner, for instance, a user may only reward good behaviour (explicit feedback), in which case, the agent can assume that a lack of reward (withheld or implicit feedback) is a punishment, and vice versa. They show that allowing the agent to deduce these "training strategies" speeds up the learning. We could imagine a parallel with our work. In our case, the agent could be "punished" for the corrections the user makes. However, the agent could also take into account a withheld or implicit feedback, such as opening a document without correcting it, as a positive reward. We discuss the implications of this with respect to how we model the user feedback in section 6.3.

The agent can also participate actively in the improvement by presenting the user with choices, or by building up models of the user's behaviour and preferences through human-system interactions to provide personalised systems. In [START_REF] Veeramachaneni | AI2: Training a big data machine to defend[END_REF], for example, the agent analyses a large raw data set, and uses outlier detection to present the analyst with a small set of events which it thinks are potential terrorist attacks. The analyst gives feedback in the form of labels for these events. The agent then uses these labels to learn or modify the models which it applies to the data set to predict the attacks.

Task performance can also be improved by asking the human to rank outcomes. This is the subject of preference-based reinforcement learning (PBRL), which combines reinforcement learning and preference learning (see [START_REF] Fürnkranz | Preference learning (dagstuhl seminar 14101)[END_REF] for a recent state of the art). The user is asked to define preferences over actions from a given state based on simulations, or rollouts [START_REF] Busa-Fekete | Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm[END_REF][START_REF] Fürnkranz | Preference Learning[END_REF][START_REF] Cheng | Preference-based policy iteration: Leveraging preference learning for reinforcement learning[END_REF]. [START_REF] Fürnkranz | Preference-based reinforcement learning: a formal framework and a policy iteration algorithm[END_REF] give two main approaches to representing preferences. The first is numeric, or quantitative, and is the standard approach in RL, leading to an estimation of the value of choosing an action in a state. The disadvantage of this approach is that the user must weigh CHAPTER 2. RELATED WORK up all the criteria composing a state, and give it a numerical value. The second approach, preference learning, is qualitative and relies on the user expressing a binary preference over possible actions, rather than having to give a concrete value.

Another approach is that of [START_REF] Akrour | Preference-based policy learning[END_REF][START_REF] Akrour | APRIL: Active preference learning-based reinforcement learning[END_REF] who ask the user to rank policies as demonstrated by the robot. They then use this qualitative feedback to optimize the policy. They point out that care must be taken to limit the complexity of the comparisons. [START_REF] Akrour | alized Feedback: Beyond Numeric Rewards[END_REF] reduce the complexity of the choice by only presenting trajectories to the user, who is asked to express pairwise preferences. They also examine the problem of trust between the agent and the human. If the agent consistently presents trajectories that are difficult to compare, then the human is likely to become inconsistent, and the agent loses trust in them. On the other hand, if the agent is well-taught at the start and presents useful information to the human, their error-rate decreases, and the agent's trust is gained. Similarly, [START_REF] Wilson | A bayesian approach for policy learning from trajectory preference queries[END_REF] present trajectory snippets to the experts (rather than whole solutions). They calculate the distance between these trajectories by comparing the state-action pairs composing them.

The issue of trust is especially important when the human and the agent have different agendas. In [START_REF] Azaria | Strategic advice provision in repeated human-agent interactions[END_REF], an agent and a human interact. The agent provides advice to the human user as to the choice he should make at each step. They show that a completely self-interested agent will lose the trust of the human, as the actions that it recommends will not accord with the human's goals. The human will then pursue his own agenda at the cost of that of the agent. By taking into account the user preferences when recommending an action, the agent can persuade the human to take a path which is beneficial to them both.

The knowledge gained by an agent can often be applied to other situations, for example, if I learn how to make carrot soup, then I can probably generalise my knowledge and make courgette soup too. This is the idea behind transfer learning; that in learning to perform one task, I gain experience which can be applied to other related, but different tasks. [START_REF] Taylor | Transfer learning for reinforcement learning domains: A survey[END_REF] give an overview of transfer learning applied to reinforcement learning. Humans can guide this experience transfer, or the machine can learn it by itself.

We saw that the agent can ask the human to rank outcomes, trajectories, or trajectory snippets. Ranking can also be done on the transition probabilities and rewards [START_REF] Bonet | Qualitative MDPs and POMDPs: An order-of-magnitude approximation[END_REF][START_REF] Sabbadin | A possibilistic model for qualitative sequential decision problems under uncertainty in partially observable environments[END_REF][START_REF] Epshteyn | Qualitative reinforcement learning[END_REF]. This aids in knowledge transfer, as we can abstract over states [START_REF] Reyes | Abstraction and refinement for solving continuous markov decision processes[END_REF], which enables us to make generalisations such as "a turn is easier to take at lower speed" [START_REF] Epshteyn | Qualitative reinforcement learning[END_REF] or "a translation introduces noise into the text".

Of course, how the agent learns might depend on the user and their training methods. For example, [START_REF] Knox | Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and performance[END_REF] focus on training an agent based on the polarity of the trainer ("encouraging", "punishing", etc.). Humans are naturally biased towards giving a positive reward, which favours myopic learning (where the agent prefers immediate gain to trying for larger, long-term rewards), but by converting episodic tasks to be continuous, the agent can successfully learn non-myopically. They differentiate between the task objective, which is the goal as seen by the user, and the learning objective which might be "to find a behavioural policy that maximises the expectation of the sum of the future human reward". They try to find the learning objective which allows the agent to perform well with respect to the task objective, in other words, they try to balance the objectives such that the agent behaves as the trainer intended. [START_REF] Knox | Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and performance[END_REF] also give an interesting insight into how to adjust the long-or short-sightedness of the agent with regard to the rewards it expects. Should it be patient, hoping for larger rewards later, or should it set out to get the maximum gain it can straight away? The parameter that affects this is known as the discount factor γ. They claim that a higher value of γ renders the agent more robust to environmental changes that could block the MDP-optimal path, for example, whilst leaving the goals unchanged. In our case, this could be a run of documents that have no extractable events, for example, or that do not receive user feedback. At the other extreme, setting γ to zero "reduces reinforcement learning of a value function to supervised learning of a reward function". The disadvantage being that the reward function represents the optimal policy, but not the task goals, and is therefore not necessarily obtainable in "real-life". This forces the trainer to micro-manage, and is clearly unacceptable in our case. Their findings suggest that we should set γ to be high, as we want the agent to be task-focused.

Finally, in production, there will not just be one user of the system, but many, and we must take all of their feedback into account. [START_REF] Shelton | Balancing multiple sources of reward in reinforcement learning[END_REF] discusses the difficulties involved in doing this, for example, how do we ensure that the users all use the same scale, what happens if the users do not agree on the feedback to be given for a particular situation, etc. In fact, our qualitative approach should help respond to some of these questions as there is no longer any need to normalise the scale used, and the definition of the high level requirements should encourage a consensus to be reached. We first present the challenge that we faced in the context of the CIFRE contract, that of extracting information from open-source documents for the Open Source INTelligence (OSINT) community using the platform WebLab, and explain a little more about the OSINT community.

We then give a rapid introduction to reinforcement learning (RL) and Markov Decision Processes (MDPs), in order to make our implementation choices clearer.

We go through the thought process of modelling a chain as an MDP, starting with a generic discussion of the information available, which should be applicable to any other chain. We then go into the specific choices that we made for our implementation.

We present BIMBO, the brains behind the operation. She is a fully configurable, modular platform, applicable to a variety of situations (we later apply her to image analysis in chapter 13, for example). She is responsible for interpreting the AIs' abstract choice of an action into a tangible one, and translating the current state of the document and system into a state that the AI understands. She measures the quality of the results found, keeps track of the time spent on treatment, and outputs logs and result files in order that the performance of the AI can be monitored.

We give a first suite of experiments which act as a proof of concept, showing that modelling an IE treatment chain as an MDP and then using a standard reinforcement learning technique to build the chain step by step is feasible.

We then move away from the standard approach of rewarding the agent with quantitative, or numerical feedback, and discuss how it could be rewarded with qualitative, or non-numerical feedback.

Finally, we implement a new algorithm, SSB Q-learning [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF], and carry out a variety of tests that show that qualitative (nonnumeric) feedback can be given with excellent results.

The industrial challenge

As we said in section 1.1, our research is driven by the Open Source IN-Telligence (OSINT or OSCINT) community. The aim of OSINT is to extract structured information from unstructured open-source data. [START_REF] Steele | The importance of open source intelligence to the military[END_REF] defines OSINT as "[. . . ] intelligence derived from public information -tailored intelligence which is based on information which can be obtained legally and ethically from public sources."

This information contributes to the knowledge of the OSINT analyst, and enables a full appreciation of the situation, allowing informed decisionmaking and actions. These decisions and actions depend on the domain's objectives, whether it is to seek an industrial advantage, to gather technological intelligence, or to enhance an online reputation. Even though the objectives may vary, every OSINT domain faces the same two main challenges (WebLab, 2016a):

Unstructured and complex data. There are a huge number of open source data readily available, especially on the Web, and in most cases, the content is very rich. However, it is a complex and fastidious task to gather usable information. The Web is dynamic, constantly growing and changing, the sources are multi-lingual, and are presented in many different media formats. Finally, there is often little or no cross-referencing or validation of the data.

Choice of tools. Not surprisingly, with the growth in data, many tools have been developed to try to extract the information. First, the best tool for each function must be chosen, whether it is the transcription of a video, or the translation of a document. Then these tools must be made to work together so that, for example, the video can be collected and transcribed, the transcription obtained can be translated, and the relevant information extracted from the translation. Like the Web, the tools are not static. The system must therefore be flexible, allowing treatments to be adapted, sources to be changed, and different information extraction requirements to be satisfied. Many data analysis systems therefore rely on a distributed architecture, 24 CHAPTER 4. THE INDUSTRIAL CHALLENGE allowing a modular treatment of the multimedia (for example, [START_REF] Bibliography Ogrodniczuk | Linguistic Processing Chains as Web Services: Initial Linguistic Considerations[END_REF] give an overview of some processing chains that are provided as web services).

The platform WebLab

Our industrial application uses the open-source platform WebLab (2016c) for economic, strategic, and military surveillance. The WebLab platform addresses the two problems above, enabling coherent and flexible systems for OSINT to be built. WebLab integrates web services which are designed to be used in a modular fashion. They can be parametrised and chained together, interchanged or permuted to create a treatment chain for the analysis and extraction of information from web pages and other open-source multimedia. These open-source multimedia are crawled by the information retrieval (IR) tools (which are out of the scope of this thesis) and fed continuously into the chain of distributed web services which carry out modular treatments, such as transcription, translation, extraction and presentation of the results. Each service analyses the contents of the resource it receives and enriches it with annotations (see Figure 4.1). Finally, the results are stored for the analyst to consult.

Every time WebLab is used in a project, the services are tailored to suit the specific user domain, e.g. maritime, technological intelligence, etc. As a consequence, there are already many different proprietary versions of the services which could be used as "black boxes" in the treatment chain.

A typical WebLab document treatment chain (Figure 4.2) is based on open-source services available from WebLab (2016b).

• The document is passed to the normaliser [START_REF] Tika | Apache Tika -a content analysis toolkit[END_REF] which as well as converting it to XML format, also uses the Apache Tika™ toolkit to detect and extract metadata such as its format, length, file name, date of collection, etc. and the structured text content.

• The resource is then passed to the language detector NGramJ (2015), which uses ngrams of characters to determine the language of a character sequence. Confidence scores between '0' (the text is definitely not written in this language) and '1' (the text is definitely written in this language) are returned for each portion of text.

• Next, an extractor encapsulating GATE (2016) extracts the named entities.

Figure 4.1 -A multimedia source is broken down by its contents -text, image, audio and video. Annotations are added to each part and sub-part, as well as to the resource as a whole.

• Finally, the results are stored in a database accessible by the analysts.

Text

Normaliser (e.g. [START_REF] Tika | Apache Tika -a content analysis toolkit[END_REF])

d → r 0 ; r 1 = r 0 + u 0
Language detector (e.g. [START_REF] Ngramj | NGramJ, smart scanning for document properties[END_REF])

r 2 = r 1 + u 1
Extractor (e.g. GATE ( 2016)) Example 1 Consider the following document: 4/29/1971: In a series of two incidents that might have been part of a multiple attack, suspected members of the Chicano Liberation Front bombed a Bank of America branch in Los Angeles, California, US. There were no casualties but the building sustained $1 600 in damages.

r 3 = r 2 + u 2 DB d r 1 r 2 r 3
The simplified XML resource in Figure 4.3 is produced after passing the document through a normaliser, adding the annotations "text/plain" (line 12) and original content (line 19); and a language detector, adding the language "en" (line 16).

Who are the analysts?

So far, we've written about "the analysts" as though they were a single, specific group of people using the same system. In fact, they come from many different walks of life, and use many variations of the application which have been developed specially for them. The OSINT community is large and widespread, and their requirements varied. We can, however, describe a "typical" OSINT analyst here.

Firstly, they seek information from open-source multimedia to build a picture of a current situation. They will then recommend actions based on this information. With the advent of the internet, huge volumes of open source multimedia are freely available, and the analyst has to find pertinent information within the wealth of data.

This information can be extracted in a number of different ways, but depends, obviously, on the quality of the source documents (which depends on the quality of the IR tools). Extractions can therefore be erroneous, contradictory or missing. The analysts are therefore typically forced to correct a posteriori the extractions using their expert knowledge and other reliable sources to fill in the gaps. This is a distracting, onerous and repetitive task, and although these corrections are stored in the database, they are currently never fed back into the extraction process. This wastes, not only the analyst's time, but also their knowledge, as the system never learns from it.

The way in which the extracted information is presented to the analyst varies depending on the user domain and their particular system. In the open-source system, it is in the context of the original document. In Fig- Unfortunately, the current open source interface is not adapted to present the extracted events to the user. Ideally, the interface would present the information gathered as an event summary over its four dimensions -conceptual, temporal, spatial and agentive. These summaries would be indexed by 4.3. CHALLENGE SUMMARY 29 each of those dimensions, allowing the analyst to easily retrieve events that they were interested in, for example, of just one type (with the same conceptual or semantic dimension) over a certain time period, or which involve a certain agent.

Neither is an open-source fusion service available to aggregate event information gleaned from several documents. Again, ideally, if such a service were made available, each event summary should contain contextual links to the original documents from which the information was obtained, so that the analyst could easily refer back to the sources.

Given such an interface, the analyst would be able to edit the event by adding, changing or deleting part or all of the information for one or more of the dimensions. For instance, they might complete a partially extracted date, remove an incorrect geographical reference, or correct the name of one of the agents. They would also be able to correct errors made by the fusion service by merging two or more events, or by separating a single summary sheet into several events. Once they were sure that the event summary sheet, or part of it was correct, they would validate it. The analyst would also be able to see at a glance which fields had already been validated (by them, or another analyst).

Behind the interface would be an action tracker, registering, for each event, what changes were made by the analyst, which information was validated, or even if no changes were made after a summary sheet was consulted. This action tracker would link those actions (or lack of them) to the document, and hence to the treatment that the document received. This would enable feedback to be collected on the quality of that particular treatment.

The development of this ideal interface is outside the scope of this thesis, but we discuss the mechanics that we might put behind it in the following. In section 7.6, we go into more detail as to how we chose to interpret edits as a quality measure. We discuss in section 6.3 how we might use the analyst's actions to reward the AI, and in chapter 8 and chapter 10 we describe how we implemented this feedback in our tests.

Challenge summary

Although we presented a very simple chain above, in production the treatment chain is complex. It is written and calibrated by experts who choose the services making up the chain, their order, and their parameters. The service order is fixed, but can be conditional, (e.g., "if the document is in English, send it to event-extractor, and to translator otherwise"). The system experts CHAPTER 4. THE INDUSTRIAL CHALLENGE can offer the analysts some help by fixing errors in the web services, or by developing new ones to fulfil new requirements. They can also change the chain in consultation with the analysts. Despite their joint expertise, it is difficult to create the perfect universal chain, as the treatment of open-source documents gathered from the web means that: their format and contents are not standard, the source pages themselves are not controlled, URLs change or are pirated, and there is "noise" (e.g., adverts). The right web services have to be called in the correct order and supplied with the best parameters, and even then may or may not extract useful information from a document.

Undetected, and partially or falsely extracted information (e.g. unconnected words in the same sentence erroneously associated) provoke errors that have to be corrected. For example, imagine that the analyst wants information on aerial bombings. We saw in section 1.1 that it is impossible to say with certainty that the word "bombardment" refers to this. The analyst can see that the page is a school blog only once the event has been extracted. Even from a specialist web page, it is possible that the word refers to the incessant questions of the journalists, and not an attack. Maybe the synonym "shelling" was used, and the event is not recognised. With these uncertainties, the experts who configure the chain try to envisage the most common situations. It is inconceivable that they construct individual chains by examining each source document.

The existing chains are therefore open-loop systems. They follow a preprogrammed set of instructions based on anticipated outcomes to reach their goal. They cannot take into account the results of their actions, nor do they have the capacity to vary these instructions in unexpected or unusual situations. This can be compared to planning a drive between London and Manchester, programming how long and how hard to press the accelerator, how many degrees to turn the steering wheel, etc., and then blindfolding the driver. This is fine if there's no other traffic, no diversions, or unexpected red lights. . . We want to remedy this situation by making the chain a closed-loop system, where the treatments are based not only on the overall plan, but also take into account current information, and feedback on the extent to which the final goal is achieved.

We want to provide a mechanism which learns to modify its behaviour in real time, continuously improving the chain, and reducing human effort by decreasing the error rate. This mechanism receives feedback obtained as the analysts consult the system output. In other words, we allow the chain to "learn from its mistakes". For instance, that if the document is a well-written scientific article in French, a translation to English before the extraction works well to make the most of the wealth of English extraction rules available, but for blogs or tweets containing slang, a direct extraction is preferable as the dictionaries are insufficient, and a translation only introduces more noise. The services, or building blocks for the chain, will thus remain a "black box" for the analyst, allowing them to distance themselves completely from the extraction process.

We need to collect the feedback non-intrusively, that is, in a way that is invisible to the analyst without impacting on his or her work. A user's expertise can be modelled by tracing his actions [START_REF] Bratko | Learning qualitative models[END_REF]. In production, these action traces could be captured through the graphical interface that we spoke about in section 4.2 which highlights the extractions in the original document. We can then base the feedback on the analystsystem interactions that currently occur. For example, an event might be corrected (e.g. a named entity might be added ), in which case we can deduce an explicit feedback based on the corrections that the analyst has made, that is, a distance between the event as corrected and that which was extracted. Indeed, our first set of tests in chapter 8 simulate this, in that the agent receives a numerical feedback as an automatically generated judgement on the quality of extractions.

The difficulty is that an event may also be not extracted, extracted in too long a time, etc. To interpret these "non-actions" as feedback, we must first answer questions such as "Is it preferable to extract something erroneously, or to miss an extraction?", "Should I sacrifice speed for quality, or vice versa?". These questions cannot be answered naturally with a numerical response; it is not easy to determine that "A partial extraction is worth 10.5 times more than a complete extraction" or "Going 2 seconds faster is worth 3.7 times more". It is therefore essential that we progress to giving intuitive feedback (chapter 9), based on naturally expressed user preferences, such as "I prefer an extraction to no extraction" and "I prefer it to be as fast as possible". The requirements can thus be gathered very easily, and can be changed if necessary in an intuitive fashion.

We are not seeking to create a clone of the analyst, but rather to capture their hard-earned skills, knowledge and expertise. This would have a double benefit. First, it would allow the system to self-improve with each human-machine interaction, thus lightening the human analysts' workload. Secondly, analysts are expensive and increasingly rare, which makes it increasingly important not to waste their time and expertise. We could aid the transfer and perpetuation of their knowledge; when one person leaves, the next can "inherit" the knowledge of the previous one.

We faced certain operational constraints when trying to formally define the feedback we require from the operator. For example, an analyst could reasonably be expected to spend up to an hour each day initially providing feedback, but only if he sees a real improvement in the results. Also, there is a risk of the analyst unintentionally introducing misleading feedback through his knowledge and expertise. For example, even if the extraction was perfect based on the information provided in the source document, if that information was faulty or ambiguous, the analyst will see and correct an error. The headline "Nice attack in July 2016" in a sports article probably means that the team in question played well that month, but to the system, it could refer to the 2016 terrorist attack in Nice, France, leading to an extracted event which the analyst would delete. Another example could be a mistake in the reported date due to a typographical error, which again, would be extracted correctly according to the information given, but which would need to be changed by the analyst. Next, we cannot ask the analysts to compare the extraction quality on two documents, as this is not part of their normal routine, and would be cognitively difficult. Finally, as we said above, the existing open-source interface is not adapted for the presentation and modification of event summaries, nor for the collection of user actions. Given these restrictions, and the operational impact of asking the users to correct our test output, in our experiments, we simulated our "user feedback".

Chapter 5

Reinforcement learning

Part of our contribution is to model the treatment of a document by the chain as a Markov Decision Process (MDP), and its improvement as a reinforcement learning (RL) problem. Before we go into exactly how we did that in chapter 6, it is probably a good idea to explain what RL and MDPs are, so that the connection can been made more easily. If you are already familiar with the concepts, then you are more than welcome to skip directly to chapter 6. . . If you are still reading, then please note that although we try to give a self-contained overview below, we highly recommend reading [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] for an in-depth introduction to the topic.

Although reinforcement learning forms part of the family of machine learning, it has also been considered a very natural way to model human learning, but the assumption is usually that humans will always learn faster than machines. Doshi-Velez and Ghahramani (2011) refute this theory, and make an interesting comparison of human and agent reinforcement learning in partially observable domains. They found that agents are more tenacious, and have more endurance than the human learners. This was shown to give them an advantage in learning, as they are not discouraged by early failure, and persist in refining their solutions from good enough to perfect. [START_REF] Shteingart | Reinforcement learning and human behavior[END_REF] support this finding, claiming that human learners cannot have a precise and exact view of the states and actions comprising their environment. They are likely to over-complicate the scenario, make false assumptions, ignore instructions and even develop superstitious behaviour. This is one of the difficulties that we face in trying to define the environment for the agent -making it specific enough that the agent can infer, correctly, the causes and effects.

One of the most common scenarios in RL is that an agent tries to learn from trial and error interactions with its environment what behaviour is optimal. In other words, it is not given an explicit solution to a problem, but must use observations of its environment to establish how it should act in different situations to maximise a reward over time. The set of situations or states and the appropriate action to take in each forms a "strategy", or 34 CHAPTER 5. REINFORCEMENT LEARNING "policy". This optimal set of one action for every state can be hard to find, not least because the effects of an action may have short-, mid-and longterm effects. For this reason, the gains are often estimated as an expected cumulative reward -"If I do this now, I don't get an immediate reward (in fact, it may cost me something), but I can see that in the future, if I follow this plan, then I should probably get a larger reward".

This word "probably" is important. The actions that the agent takes do not lead neatly from one state to another in the same way each time (otherwise it wouldn't be interesting), instead, they are stochastic. That is, from a given state A, if the agent performs a given action, then maybe he'll end up in state B, maybe state C, or maybe he'll just stay in A. He therefore has to try to learn with what probability he'll move to a given state, and use this when calculating his reward.

Given that actions may cost the agent (he might use fuel, for example, or be on a time-critical mission), he also has to weigh up the benefit of going for immediate rewards against the cost of trying for higher rewards later.

We can summarise this in Figure 5.1. The learner receives rewards, based on the results of its chosen actions. The closer the results are to the goals, the higher the reward. The learner tries to maximise these rewards, typically by exploiting current knowledge to continue to receive good rewards, or by exploring new actions with the hope of obtaining even better ones. It chooses its actions based on its observations of the environment.

Environment

Agent

Action (explore/exploit) Observation Reward

Goals

Figure 5.1 -The agent acts according to its observation of the environment, and receives a reward based on how close it is to its goals.

Markov Decision Processes

Reinforcement learning is generally formalised as a Markov Decision Process (MDP, [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]), which models the environment in terms of states, in which actions are possible, leading to other states stochastically. The fact that the environment is in a given state at a certain instant bestows an immediate reward on the learner (or agent). The agent's objective is to choose actions such that it maximises its expectation of cumulated rewards.

Definition 2 (MDP) A Markov Decision Process (MDP) is a quintuplet (S, A, P, R, γ) where:

• S is a (here finite) set of possible states in the environment;

• A is a (here finite) set of actions (available to the agent);

• P is a set of distributions {P a (s, •) | s ∈ S, a ∈ A} ; P a (s, s ′ ) is the
probability that the environment is in state s ′ after the agent performs action a in s;

• R is a reward function, defined on the states; R(s) is the reward obtained by the agent when it reaches state s;

• γ ∈ [0, 1] is a discount factor, weighting expected future rewards against those currently expected.

In RL, the agent initially only has knowledge of the state / action space S ×A, as well as the factor γ. The probability distribution is unknown, which means that it has to learn to "navigate" between the states. For instance, from a given state s 0 , if it takes action a 0 ten times, it might arrive in state s 1 3 times out of 10, in state s 2 6 times out of 10, and in state s 3 only once. This means that it would learn that P a 0 (s 0 , s 1 ) is 0.3, P a 0 (s 0 , s 2 ) is 0.6 and P a 0 (s 0 , s 3 ) is 0.1. Note that this knowledge only reflects the agent's experiences, and not necessarily the real probability function -it may just have got lucky (or not). It also has to find out what reward is given for each state, i.e. the reward function, and which path of actions it therefore expects will probably give the best results.

More formally, at each instant t, it knows the current state s t of the environment, and chooses an action a t . The environment passes into state s t+1 according to the probability distribution P at (s t , •), and the agent is informed of the state s t+1 and the reward r t+1 = R(s t+1 ). The process continues in s t+1 .

The agent, as it interacts with the environment, learns a series of policies π 0 , π 1 , . . . , π t , . . . , where a policy π t : S → A gives, at instant t, the action π t (s) to perform if the current state s t is s. Its goal at each moment is to maximise the expected accumulated reward, where the expectancy is taken over the trajectories generated by the policies π 0 , π 1 , . . . , π t , . . . , that is, the expected quantity ∞ t ′ =t γ t ′ R(s t ′ ). This generic framework encompasses many variations (for a recent summary, see [START_REF] Szepesvári | Algorithms for reinforcement learning[END_REF] P a 0 (s 0 , ?) =? P a 1 (s 0 , ?) =? Example 3 As a tiny example of how we might model a process as an MDP, we sent a robot shopping for grapes and milk. Figure 5.2 shows what he learnt the first time, and Figure 5.3 what he found out the second time.

• Figure 5.2a: He starts in state s 0 with an empty basket, and the choice of two actions -put the milk in the basket (a 0 ) or put the grapes in the basket (a 1 ). He has no knowledge yet of what the consequences might be of each action, as he has currently no idea of the probability distribution or the reward function.

• Figure 5.2b: He chooses a 0 randomly, and puts the milk in the basket. He's now in state s 1 , i.e. there's just milk in the basket. He hasn't finished the shopping, so isn't rewarded yet. He now knows that the probability P a 0 (s 0 , s 1 ) is 1.0 and the reward R(s 1 ) is 0.0.

• Figure 5.2c: From s 1 , he puts some grapes in the basket (action a 1 ), and is now in state s 2 : milk under grapes, and now P a 1 (s 1 , s 2 ) is 1.0.

• Figure 5.2d: He receives a reward of +5, and adds to his knowledge that R(s 2 ) is 5.0 (depending on the RL algorithm used, some of this reward might be back-propagated, so that he can update R(s 1 ) based on a proportion of 5.0).

• Figure 5.3a: Next week, we send him shopping again. He starts in state s 0 with an empty basket, and the same choice of a 0 or a 1 .

• Figure 5.3b: He chooses action a 1 this time, and puts the grapes in the basket. He's now in state s 3 , with just grapes in the basket. As before, he hasn't finished the shopping, so isn't rewarded yet. He now knows that the probability P a 1 (s 0 , s 3 ) is 1.0 and the reward R(s 3 ) is 0.0.

• Figure 5.3c: From s 3 , he puts some milk in the basket (action a 0 ), and is now in state s 4 : milk on top of grapes, and now P a 0 (s 3 , s 4 ) is 1.0.

• Figure 5.3d: Bad robot! The milk has squashed the grapes, and there's grape juice running everywhere, so he receives a punishment of -100, and adds to his knowledge that R(s 4 ) is -100.0.

Of course, the probability P a 0 (s 0 , s 1 ) is not really 1.0. Maybe 10% of the time he drops the milk on the floor, which would put him in state s 5 (milky floor). In this case, after a lot of shopping expeditions to remove the effect of chance, he should learn that P a 0 (s 0 , s 1 ) is 0.9 and P a 0 (s 0 , s 5 ) is 0.1. Finally, we hope that he would learn the policy π(s 0 ) is a 0 and π(s 1 ) is a 1 , i.e. to always put the milk in first, followed by the grapes.

Q-LEARNING

Q-Learning

Numerous algorithms exist to solve RL problems, and one of the simplest to understand and use is called Q-Learning [START_REF] Watkins | Learning From Delayed Rewards[END_REF]. The few parameters that it takes can be set intuitively, and the results can be observed and interpreted easily. It was for this simplicity that we chose it for our proof of concept tests. We wanted an algorithm where any external side-effects, for example due to unfortunate parameter choices, were reduced to a minimum. We give a brief overview of the Q-Learning algorithm (Algorithm 1) here to keep this document self-contained, but encourage you to read the original article.

Algorithm 1: Q-learning Data: MDP M 1 while True do 2 Choose a t in s t using the EG exploration strategy 3 Play a t , observe s t+1 , and let

r t+1 = R(s t+1 ) 4 Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )(r t+1 + γmax b { Qt (s t+1 , b)} -Qt (s t , a t ))
Q-learning maintains, for each state / action pair (s, a), a value denoted Q(s, a) which represents the agent's current estimate of the expected cumulative reward if from s it executes a, then follows an optimal policy.

It is controlled using an ǫ-greedy (EG) strategy: when the agent is in state s t at time t, it chooses a "greedy" or optimal action a t which maximises Q(s t , a t ) (it exploits), except with probability ǫ, when it chooses a random action (it explores). Then, upon observation of the new state s t+1 and immediate reward r t+1 , it updates Q(s t , a t ):

Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )(r t+1 + γ max b { Qt (s t+1 , b)} -Qt (s t , a t ))
where:

• the learning rate α ∈ [0, 1] fixes the importance of the latest experience (r t+1 +γ max(. . . )) with respect to the experience already gained (that is, the previous value of Q(s, a));

• the discount factor γ ∈ [0, 1] is as in Definition 2 of an MDP on page 35, i.e. it determines the importance of long-term gains in comparison to short-term ones.

Other applications of reinforcement learning

One of the currently blossoming fields in reinforcement learning is that of "deep reinforcement learning". Popularised by Google DeepMind team [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], they have already applied it successfully to learning how to play Atari Games [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF]. They designed a variant of the Q-Learning algorithm described above, called Deep Q-Learning, and approximated the Q-values (which they refer to as the "action-value" or "Q-function") with a neural network. Of course, reinforcement learning is not just applied to silly examples such as robots going shopping, or playing 1980's video games. It can be applied in much more complex situations, where the environment observed does not just depend on the actions taken, but also on external circumstances, and other agents' actions.

For instance, INVESTOPEDIA (2016) defines algorithmic trading (automated trading, black-box trading, or algo-trading) as "the process of using computers programmed to follow a defined set of instructions for placing a trade in order to generate profits at a speed and frequency that is impossible for a human trader."

If you buy or sell shares, then you have an impact on the market. Very simplistically, and making gross generalisations, if you buy a huge amount of a given share, then you should push its price up because people think that if you want that many, you obviously know that those shares are good. Conversely, if you sell a vast quantity, then the price will fall, as everyone will panic and start to sell their shares in a snowball effect. Assuming that the world is constant around you, then you should be able to manipulate the stock market by buying and selling to make yourself a profit. The problem is, not only are other people trying to do the same thing, but there are also external factors which influence the share price. A concern that a product causes cancer would certainly make shares in that product's manufacturer fall. The announcement of breakthrough in reducing production times will probably bounce the price up. By using algorithmic trading, companies hope to benefit from the buying / selling effect, without the annoying side-effects caused by the outside world.

Of course, the algorithms are not just fixed sets of instructions, normally there is a learning process behind them, and for those of you looking to make your fortunes by playing the stock market using Deep Q-Learning, [START_REF] Nydal | Deep q-learning for stock trading[END_REF] gives a nice example of how to use reinforcement learning on existing stock market data.

Chapter 6

Modelling a chain as an MDP

Given that (as we saw in section 4.3) it is impossible to create a unique chain capable of perfectly treating every type of document, the ideal would be to have a tailor-made chain for each one.

Formalising this treatment chain and improving the extractions, however, is not a trivial problem. The only information known before starting the treatment chain consists of the available services, their parameters, and the potential characteristics of the document, XML resource and system (see chapter 4). The agent knows neither the form, nor the content of the documents in advance. The ideal treatment for each document is unknown and must be learnt. For example, it depends on the language in which the document is written, its source, etc., which are only discovered during the treatment. It does not even know the proportion of documents written in each language, coming from each source, etc. As with the "milky floor" in Example 3, it can only learn these proportions through treating many documents. It does not know if the document contains information nor even if an extraction is possible with the given tools. Neither does it know the interfaces (inputs and outputs) of the services. Its actions are thus taken under uncertainty. As we saw in chapter 5, this sounds rather like the definition of reinforcement learning and MDPs. Part of our contribution, therefore, is to model the treatment of a document by the chain as a Markov Decision Process (MDP), and its improvement as a reinforcement learning (RL) problem. We explain first how we modelled the states and actions, and then we tackle the thornier problem of how to reward the agent.

Choice of states

Our initial ideas for state attributes were inspired by the possible annotations that are added to the WebLab resource by the services in the treatment chain (see Table 6.1). Each annotation contains information about the document, and about the extracted events and named entities. As the time taken to process a document is also important, we wanted to include system information such as time taken in the definition of a state too.

CHAPTER 6. MODELLING A CHAIN AS AN MDP

Annotation

Description and example refersTo Gives the type of the event or named entity and its unique reference identifier (uri), e.g. DeathEvent#1 label

The plain text of the event or named entity (the subject of refersTo), e.g. member of HAMAS name

The geonames location of the event, e.g. The identified language of the input document, e.g. en hasOriginalFileSize File size of input document, e.g. 339 format Format of input document, e.g. text/plain Table 6.1 -Potentially useful annotations to use when defining the state characteristics.

CHOICE OF STATES

We therefore defined the states using the characteristics of the original document and of the resource and system at a given moment. The system thus perceives the task as the states of a process, and each passage through a service modifies the current state.

More precisely, the states are formed from the values of a certain number of descriptors of the documents and of the system. The states that the system perceives are thus combinatorial states s i which can be defined at time t in terms of the features of the resource and the current state of the treatment chain. If we say that the document and resource properties are represented by the features X i and the system properties by the features Y i then we could represent some states as in Table 6.2. Table 6.2 -A possible representation of states given document features X i and system features Y i , where " " indicates that the information is not (yet) available. For instance, X 1 is the detected language that the document is written in; X 2 is the source of the document; X 3 indicates whether the type of document has been extracted; X 4 is the type of document; Y 1 is the time taken so far in seconds; Y 2 is the number of services already called, Y 3 is the number of parameters changed.

X 1 X 2 X 3 X 4 Y 1 Y 2 Y 3 s 1 en
These states allow a generalisation of the learning; we have already seen that the type of source web page (school blog vs. specialist) could greatly influence the utility of the word "bombardment" for the extraction of information. We might hope that the system would learn from the user interactions that:

• if the current state has the value "true" for the descriptor "typeExtracted" and the value "school blog" for the descriptor "type" (as in s 1 in Table 6.2), then the best action to perform consists of stopping the treatment (useless to try to extract bombings from school blogs);

• if the type is extracted and is not "school blog" (as in s 2 in Table 6.2), the best action consists of passing the document to an extraction service, using the word "bombardment" among the trigger words;

• otherwise (as in s 3 in Table 6.2), the best action consists of passing the document to a type-recognition service.

Obviously, the agent will never "see" the resources nor the documents. Rather, it will see a document transformed through a treatment chain as a series of projections of resources, that is, as a series of vectors of values. If we want the agent to be able to generalise, then it is very important that the features are abstract enough for similar values to occur with a reasonable frequency, but not so abstract that they do not help the reasoning. For example, a resource feature indicating whether the document contains the name "Elizabeth II" would certainly be too precise, whereas one indicating that the document contains text would be too vague. A resource feature indicating that the document contains the name of at least one monarch, on the other hand, could be more useful.

An astronomer, a physicist and a mathematician are on a train in Scotland. The astronomer looks out of the window, sees a black sheep standing in a field, and remarks, "How odd. All the sheep in Scotland are black!" "No, no, no!" says the physicist. "Only some Scottish sheep are black." The mathematician rolls his eyes at his companions' muddled thinking and says, "In Scotland, there is at least one sheep, at least one side of which appears to be black from here some of the time." [START_REF] Stewart | Concepts of Modern Mathematics[END_REF] The features may be directly read from the annotations (e.g. the language detected), or may be derived (e.g. the number of royal family members identified). We also wanted to include a feature which could be defined by the analyst in order that he receives personalised extractions, for example, the type of event that they are particularly interested in.

We therefore decided to represent a state by an attribution of the following characteristics:

• the information already extracted (if any), where " " indicates not (yet) extracted:

language ∈ {"en"," "};

format ∈ {"text/plain"," "};

interesting ∈ {true, false} (true if at least one event of the type that interests the analyst has been extracted);

any ∈ {true, false} (true if one or more events of any type have been extracted);

• the parameters chosen (see section 6.2 below for an explanation):

-nounGazetteer (the noun gazetteer currently chosen, if any);

-verbGazetteer (the verb gazetteer currently chosen, if any);

• the system state:

-nbServices ∈ {0 -5, 6 -20, 21+} (the number of actions the AI has already taken, i.e. the number of services through which the current document has already passed plus the parameter changes. We explain in section 6.2 just below why these ranges were chosen);

-timeTakenSoFar (number of seconds since the current document treatment started in timeInterval (parametrisable) steps).

Obviously, with access to the appropriate web services able to extract the information, more characteristics could be taken into account, such as

• document metadata: length, number of characters, number of paragraphs, number of words, number of pages, number of images, year, keywords, bibliography, citation, complete list of languages;

• document quality: syntax quality (e.g. number of spelling mistakes), complexity, subjectivity;

• source type, e.g. tweet or thesis, popular or specialised;

• theme, e.g. sport, science, philosophy.

It should be noted that the features that we detailed above were chosen in the context of the CIFRE contract, given the industrial application that we use. However, the system is completely modular, and not at all dependent on a given state definition as we show, for example, in chapter 13 where we define a state in terms of the parameters of image analysis services.

Choice of actions

The system also has a certain number of actions available which it can apply in the current state. A natural choice is to offer the AI the choice of the next service to call, or to stop the treatment and offer the results to the analyst. This reinforces our choice of an MDP to model the treatment, as it takes into account probabilistic uncertainty (the results of calling a service on a resource are CHAPTER 6. MODELLING A CHAIN AS AN MDP not known for certain in advance), and the services take the system from one state to another. For example, if 70% of the source documents are in English (and the language extractor is completely reliable), the agent will perceive that taking the action "detect language" in a state s t leads with probability 0.7 to a state s t+1 similar to s t except that it contains the information "language detected" and the annotation "en".

However, building an efficient treatment chain depends not only on choosing the services but also on setting their parameters correctly. For instance, the extractor service encapsulating GATE (see section 4.1 on page 24) that we use relies on gazetteers, which are lists of nouns and verbs triggering the detection of a specific type of event (e.g. the bombing verb gazetteer contains "bomb", "explode", "detonate", etc.). GATE is therefore a parametrised service; choosing the correct gazetteers for it to use is part of building an efficient treatment chain, as without them it is ineffective.

One of our initial ideas was to dynamically construct these gazetteers using online thesaurus services. This proved technically problematic for two reasons. First, the online services restrict the number of queries made on them, which would have significantly reduced our test capabilities. Second, the gazetteers contain not only the event detection trigger words, but also the grammar rules (JAPE ( 2016)) applicable to each word. These rules are difficult to construct, even for a native speaker, let alone an AI [START_REF] Chiticariu | Rule-based information extraction is dead! long live rule-based information extraction systems![END_REF]. We therefore took the gazetteers already defined in production.

The available actions are therefore to choose the next service from {Tika, NGramJ, GATE }, to choose a GATE gazetteer, or to STOP the treatment and return any extracted events. This means that in an optimal chain, the AI would choose two gazetteers (one list of verbs, and one of nouns) plus three services (Tika, NGramJ and GATE ) giving five actions, before choosing to STOP. The ranges given for nbServices in the state therefore reflect an "optimal" number of actions (0-5), a "slightly sub-optimal" number of actions (6-20) and "far too many" actions (21+).

Note that we treat the services as "black boxes" so that knowledge of, e.g., their WSDLs (Web Service Definition Language) is unnecessary. Of course, any such extra information could be used to increase the efficiency and robustness of the learning process and we discuss this in section 14.2: Future work.

The repetition of the same action on the same document is technically authorised, but will penalise the system, as the treatment time will be longer, and due to the massive volumes being treated in production (potentially all possible documents from the web) faster results are preferred.

Rewarding with user feedback

Solving an MDP with RL requires that the agent be rewarded for its actions. The closer it is to achieving its objectives, the higher the reward. In our case, only the analysts can specify the objectives, and hence the reward, using their skill and expertise to judge how close the agent came to perfection. In other words, their feedback on the treatment of a document will reward the agent, and reinforce its behaviour.

In section 2.5, we saw that [START_REF] Bratko | Learning qualitative models[END_REF] raised the question of reconstructing the skill of an operator.

They highlight the difficulty of engaging the operator in a dialogue, trying to formalise his or her skill. The operators are usually not capable of describing exactly what they do, and any description furnished is likely to be incomplete and imprecise. Dialogues are also intrusive; the operators already have a principal task to accomplish, and interruptions designed to get information about the reason for their actions are distracting or annoying. To keep interruptions to a minimum, the dialogues should therefore ideally be conducted on a qualitative basis in terms that are intuitive for the operator. We tackle the problem of formalising just such an informal description of a skill in chapter 9, where we examine rewards given in an intuitive, qualitative fashion, easily expressed and quickly specified by the users. [START_REF] Bratko | Learning qualitative models[END_REF] then suggest identifying the skill from the traces of the user's actions. We are not seeking to create a clone of the analyst, but we are seeking to capitalise on their expertise and judgement of what the results are worth (i.e. their knowledge of the true reward function of the MDP) to build a coherent model. Moreover, we're seeking to build a system that self-improves with each interaction, and which encapsulates the skills of each analyst with whom it interacts. This inspired our gathering of quantitative rewards through observation of the edits that the analysts carry out on an extracted event (see chapter 8). [START_REF] Bratko | Learning qualitative models[END_REF] remark that once a clone of the operator is trained, it can perform better and more consistently than the operator. This reinforces our idea of the continuous improvement of the AI constructing the chain; eventually, it should construct the chain better and more consistently than even the experts. They point out that another benefit is not just the reproduction of an operator's skill, but understanding it, and being able to transfer the acquired knowledge to less experienced operators. Indeed, a trained system benefits everyone.

Before we could tackle the modelling of the rewards, we had to ask CHAPTER 6. MODELLING A CHAIN AS AN MDP ourselves several questions. The first, and arguably the most important, is at what level we wished to invite the analysts to reward the agent. Should it be on the document as a whole, or on individual attributes within that document? Certain questions will apply to all documents (for example, the format of the document is never detected), others only to individual documents (for example, a sentence is not translated correctly). We decided that it should be a mixture. A numeric reward is given on the attributes, proportional to the number of edits that the user carries out (see chapter 8), and a higher level qualitative reward (see chapter 10) is given on the extractions within the document (and therefore the treatment) as a whole -was there an extraction, was it fast, etc. The advantage of this approach is that there is no need to separate the training phase and long-term usage. The system continues to learn with every document it treats. [START_REF] Loftin | Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning[END_REF] show us that agents can learn not only from explicit feedback, but also from withheld feedback. For instance, a user may only reward good behaviour, in which case, the agent can assume that a lack of reward is a punishment, and vice versa. We could imagine a parallel with our work. In our case, the agent is mostly "punished" by the user corrections, but the agent could also take into account a "non-feedback", such as opening a document without correcting it as a positive reward. The difficulty is that in our case, the analyst may have closed the document without correcting it, simply because he was too busy, or just not interested in the contents. We therefore considered asking the analyst to provide an explicit feedback every time they consulted a document. For instance, by choosing from the options "I'm correcting this because it's bad", "I'm adding information because it's incomplete", or "I'm leaving it alone, because it's perfect / because I don't know / because I don't have time to correct it". We also thought about allowing the AI to ask why a document was, or was not corrected. Given that our aim was to be as unintrusive as possible, we rejected these ideas, opting for a behind-the-scenes approach to gathering the rewards.

As we discussed in section 4.2, the event summary would allow the analysts to consult the results produced by the system for a given event by showing the information extracted for each of its dimensions, linked back to the original source documents. The analyst would be able to see at a glance which events or parts of events had already been validated (by them, or another analyst), and they would be able to edit, add or remove information. They would also be able to split or merge events. The corrections made by the analyst to the extracted information and captured by the action tracker would indirectly furnish a reward on the treatment the document received. The rewards r(s) are thus given to the system only for the final states of a treatment, and are defined in three ways: a quantitative value based on the corrections made to the extraction (if any) by the analyst and the document treatment time (chapter 8); a qualitative reward based on the user's purely ordinal preference on the results; and a weighted ordinal preference giving a middle-ground between the first two (chapter 10).

Chapter 7 BIMBO: a flexible platform

Introducing BIMBO

As we stated in section 1.2, our contribution consists not only of modelling a treatment chain as an MDP, but also of building a flexible platform BIMBO: Benefiting from Intelligent and Measurable Behaviour Optimisation. We built BIMBO in a completely modular fashion, allowing her elements to be changed easily (Figure 7.1 shows BIMBO's modules, which we will describe below.)

This means that adding a new reinforcement learning algorithm, or changing the definition of a state, for example, changes nothing in the rest of the platform. This modularity enables us to test easily a range of RL algorithms, input documents, similarity measures, rewards, user feedback, models, available services, etc. We later proved this flexibility by using BIMBO in a completely different context (see chapter [START_REF]<p r o p e r t y name=" morphDilateType " v a l u e="MORPH_RECT" /> <p r o p e r t y name=" k e r n e l S i z e " v a l u e=" 0 , 3[END_REF], but initially we ran our experiments with our industrial application.

In Appendix A, Listing A.1, we give an example of the configuration file which enables every aspect of BIMBO and her AIs to be controlled, from the parameters used in the algorithm to whether progress reports should be sent by SMS.

In our tests, we used a simple, but typical chain (as described in chapter 4) as a reference, with two differences:

• Firstly, to check that the AI is capable of rejecting useless services, we added the possibility of choosing a service Geo. This service is a geographical inference service, which relies on the extracted spatial dimension of the event to add extra details from the geographical database Geonames (2015). For example, if "Paris, France" is extracted, we can infer "Europe". Obviously, in production, this service could be called to complete the event summary, but it contributes nothing to the extraction process itself, and so the AI should consider it useless. We therefore used a more elaborate service which encapsulates the ontology WOOKIE (developed by [START_REF] Serrano | Vers une capitalisation des connaissances orientée utilisateur: extraction et structuration automatiques de l'information issue de sources ouvertes[END_REF], see Appendix B) and GATE, which is also capable of extracting OSINT events. This particular service uses gazetteers, or lists of trigger words for event detection in a text. These lists also contain the grammar rules (JAPE, 2016) applicable to each word. Unless otherwise specified, all future references to "GATE" in this document can be read as "the service encapsulating WOOKIE and GATE".

Recall that in section 6.2, we decided that the actions available to BIMBO were to choose the next service from {Tika, NGramJ, GATE } (we now also add Geo), to choose a GATE gazetteer, or to STOP the treatment and 7.1. INTRODUCING BIMBO 53 return any extracted events. In section 6.1 we defined the states by the characteristics:

• the information already extracted (if any), where " " indicates not (yet) extracted:

language ∈ {"en"," "};

format ∈ {"text/plain"," "};

interesting ∈ {true, false} (true if at least one event of the type that interests the analyst has been extracted);

any ∈ {true, false} (true if one or more events of any type have been extracted);

• the parameters chosen:

-nounGazetteer (the noun gazetteer currently chosen, if any);

-verbGazetteer (the verb gazetteer currently chosen, if any);

• the system state:

-nbServices ∈ {0 -5, 6 -20, 21+} (the number of actions the AI has already taken, i.e. the number of services through which the current document has already passed plus the parameter changes);

-timeTakenSoFar (number of seconds since the current document treatment started in timeInterval (parametrisable) steps).

The chain is written as a [START_REF] Camel | Apache Camel[END_REF] route in XML (see Appendix C). We defined each of the four services as an endpoint (Figure C.1). The route "consumeFile" (Figure C.2) is triggered as soon as a file is copied into the "aichain-to-process" folder. After adding the header information, such as the date and file name, it sends the WebLab resource to the Dynamic Router in "direct:start" (Figure C.3). The Dynamic Router then passes control to BIMBO. Specifically, it calls the Java method "performActionMaster". This method is responsible for informing the AI of its current state, asking it to choose an action, and then passing the control back to the appropriate part of the camel chain. For instance, if the AI chooses "Tika" (or rather, the integer corresponding to the action "call the web service Tika"), then we call the route "direct:tika", which passes the current WebLab resource to the endpoint "weblab:analyser:service-tika". If the AI chooses to change one of the gazetteers, then we copy the chosen gazetteer into the GATE folder, and call the route "direct:dummy" which does nothing (except display debugging messages if required) but pass the control back to "performActionMaster" again.

We thus give BIMBO and her AIs complete control over all the services called, their order and their parameters.

In section 6.3, we explained that in RL, the rewards that the agent is given are based on how close its results are to the goals it was set. In our platform, we cannot define these goals exactly, as they will vary from document to document. Instead, we have to reward the agent based on the analyst's feedback on their impression of the results of the treatment that a document received.

An example decision process for one document

Before we describe our platform in detail, and to better visualise the model we chose in chapter 6, here's an example of an actual document treatment taken from just under halfway through the learning process (during the tests with numeric rewards). Figure 7.2 gives a graphical representation of the states and actions, and Figure 7.3 shows the log produced by BIMBO.

The AI starts in state s 0 , from which it has already learnt that the best action is to choose a GATE gazetteer, specifically the mixed list of verbs.

BIMBO informs it that it is now in state s 84 , and has received zero reward.

The only difference between state s 0 and state s 84 is that the "Verbs chosen" property is no longer empty, but has the value "MixedVerbs".

Again, the AI exploits its current knowledge by taking the action that it has learnt is the best from state s 84 , that is, to send the document to Tika to detect its format and normalise it into an XML format.

The web service successfully detects that the format is "text/plain". Therefore, the state s 268 in which it now finds itself is the same as the previous one, but with the property "Format" equal to "text/plain".

From state s 268 , we see that there is still some learning to do, as the AI thinks that the best action is to choose the dummy list of nouns.

This takes it to state s 269 where, naturally, the value of the property "Nouns chosen" becomes "DummyNouns". In state s 194 , the AI chooses a sub-optimal action on purpose to expand its knowledge (it explores). It chooses to send the document through the service "Geo". This only wastes time as it does not extract any new information, and the AI finds itself back in the same state.

The next choice it makes is unfortunately to change the verbs gazetteer again, arriving in state s 149 . I say unfortunately, as in this particular experiment the mixed list was supposed to be the best choice. The property "Verbs chosen" becomes "BombingVerbs", and the property indicating the number of actions chosen "No. actions" is increased to the next range, i.e. to "6-20".

The AI then makes another wise choice, sending the document through the

DEFINITION AND EXAMPLE OF AN EVENT

57 language detector. The service successfully detects that the document is written in English, and at last the property "Language" is filled in. This is good news, as now all the information about the document itself that GATE needs to start extracting events is there. We just need the AI to choose the right gazetteers now, and to send the document through GATE.

After another bad choice of gazetteer (the injure verbs), the AI finally chooses the mixed verbs gazetteer again, and arrives in state s 180 .

It then chooses the shooting nouns gazetteer. Spoiler alert: this turned out not to be important, as we discovered later (thanks to the AI making choices like this) that this GATE service is faulty and erroneously never uses the nouns! This detection of errors in the services was an unexpected benefit of the platform, and could be interesting to explore in the future. For example, if a service exhibits a surprising behaviour, an alert could be raised with the development team, and another service substituted automatically.

At last, the AI sends the document to GATE. This is the most costly service in terms of time, and the AI has to learn rapidly to use it only once at the end of the chain, otherwise it is heavily penalised.

The AI is now in state s 379 , and it "knows" that it has managed to extract an event ("Any event extracted" is true), even if it was not the one that the analyst had specified ("Interesting event extracted" is false).

It therefore decides to STOP the treatment of this document.

At the end of the log, we can see that an event was extracted in 8 seconds, with a similarity of 0.6 to the oracle (more details on how we measured this are in section 7.6). At last BIMBO can reward the AI, so for this particular result, she gives it 75.0 (the similarity multiplied by 1000 for readability, and divided by the time taken). The AI will use this reward to update its estimates of the best actions to follow in each state.

Definition and example of an event

We've spoken about the events that our chain is supposed to extract, but we've not yet formally defined what they look like. As the context of our CIFRE contract means that we use the WebLab services, it was natural to use the associated definition of an event given by [START_REF] Serrano | Vers une capitalisation des connaissances orientée utilisateur: extraction et structuration automatiques de l'information issue de sources ouvertes[END_REF]. In practise, the system is modular, and with the appropriate extraction service, any other definition such as those discussed in section 2.2 could also be used.

Definition 4 (Event) An event E is a quadruplet (C, T, G, A) where:

• C ⊆ C is the conceptual, or semantic dimension, given by a set of elements taken from the domain C common to all events;

• T is the temporal dimension, or when the event occurred. It is potentially ambiguous, e.g. "last Tuesday", and to model this ambiguity, we take T ⊆ T, where T is the set of all dates;

• G ⊆ G is the spatial dimension, or where the event occurred, also potentially ambiguous;

• A ⊆ A is the agentive dimension, or the participants.

Within this general definition, we use precise domains:

• C is the set of all types of event, given by a fixed and finite set of elements in the ontology WOOKIE (see Appendix B) used by the extractor;

• T is the set of all relative and absolute "dates", such as "tomorrow", "2001", "2001/9/11";

• G is the set of entities defined in Geonames (2015);

• A is the infinite set of all extractable participants, seen as strings.

Example 5 (Extraction of an event) Recall the document that we saw in Example 1 on page 26: 4/29/1971: In a series of two incidents that might have been part of a multiple attack, suspected members of the Chicano Liberation Front bombed a Bank of America branch in Los Angeles, California, US. There were no casualties but the building sustained $1 600 in damages.

Using Definition 4, an event E = (C, T, G, A) could be extracted from this document with C = {AttackEvent, BombingEvent}, T = {4/29/1971}, G = {Los Angeles, California, US}, and A = {Chicano Liberation Front, Bank of America}. From these web pages, the treatment chain was able to extract event information, such as that shown in Figure 7.5. For the brave of heart, an example of the WebLab resource corresponding to this web page at the end of the treatment is shown in Appendix E.

Unfortunately, once the duplicates and pages which were no longer valid were removed, only 51 documents remained, which was too small to construct a decent corpus. We therefore turned to the summaries of the events (for example, Figure 7.6) to create our corpus.

These summaries are more informative, and less noisy than the original web pages. Another advantage of using the summaries is that we have one document -one event, which simplifies the evaluation of the results. The disadvantage is that they are homogeneous, being all the same format and approximate length, and written in the same language (English). This homogeneity does, however, allow us to generalise on a random sample of documents. Ideally, with access to infinite resources, we would have constructed a corpus, annotated by hand from a variety of different documents.

This set of summaries gave us documents {d 1 . . . d N } from which a perfect chain could extract perfect events E 1 , . . . , E N , respectively (as in Figure 7.6), if it had infinite time, and access to an infinite number of perfectly parametrised web services. Obviously such a chain does not exist in real life, so we use the results from a production chain constructed by hand as a reference, referred to hereafter as the "expert" chain.

We expect our AI to learn to construct chains that approach and eventually improve on the expert chain, in learning not only the correct order of the

CONVERSION OF THE CORPUS TO WEBLAB TYPE EVENTS61

services and the best parameters, but also the fact that certain services (in our case, the service geo) and certain parameters (some GATE gazetteers) are not useful.

Conversion of the corpus to WebLab type events

In order to obtain meaningful results from the AI, large numbers of documents need to be treated by the system, and the accuracy of the resulting extracted events evaluated. As stated in section 4.3, due to operational constraints, we could not ask the analysts to participate in our tests. We therefore developed an automatic comparison of events extracted by the chain with their GTD counterparts to simulate the analysts' feedback. This comparison is also a key component of the success measure for the AI for that instance of the treatment chain (see section 7.6).

There are two sets of event information: the GTD files, and that extracted by the treatment chain. These sets need to be normalised as they are not in the same format, and then compared to give a "user feedback" on each event.

Each event in the GTD files is identified by a unique reference number. There are 133 fields / columns for each event. Some of these contain information that cannot be extracted from the summary (for example the GPS coordinates) or which are not useful in our experiments (for example links to the documents or web pages which serve as source of the information). For full details, please refer to Global Terrorism Database (2016a).

Table 7.1 gives an abridged example of an event from the GTD files (note that we have based this example on the fusion of several similar GTD events in order not to leave blanks). In Table 7.2, we show the abridged event from the GTD file, corresponding to the summary shown in Figure 7.6 on page 60.

The dimensions of the GTD event are filled using both the information given directly in the GTD files, and inferred information as shown in Table 7.3:

The agentive and spatial information contained in the GTD files is entered by hand. Often, therefore, generic words and phrases are introduced which reduce the measured similarity unnecessarily. For instance, take the following example of a summary from which we would hope to extract the spatial named entities 'Ayn al-Hulway Palestinian Refugee camp and Sidon:

GTD field

Example We therefore reduced this noise by removing the following generic words and phrases from the agentive and spatial dimensions: the attack, the incident, occurred, took place, in front of, near, in, at, of, on, and, the, a, an, unspecified location, undisclosed, was / were targeted.

As in standard in string manipulation, we also removed from all the dimensions, punctuation such as quotes, fullstops, commas and hyphens. We trimmed the strings, and replaced multiple whitespaces with a single space character.

Measuring the quality of the results

In order to test whether our AI is making a difference, and whether it is "intelligent enough", we need to measure its IQ, or how intelligent it has become. In other words, to identify at what point it becomes useful or mature. We therefore need to measure the quality of its results.

To do this, we must first define the similarity between an event extracted from the document by the chain

E 1 = (C 1 , T 1 , G 1 , A 1 ), and the correspond- ing "perfect" event E 2 = (C 2 , T 2 , G 2 , A 2 ).
In our experiments, E 2 is the event as detailed in the GTD files and here we have defined a fairly simple similarity measure which is specific to the GTD -WebLab mapping. However, in production, the comparison would be taken between the event extracted and the event as corrected by the analyst. In this case, the similarity measure would certainly need to be rethought1 . For instance, when comparing with the GTD, the geographical information is either correct or not, but in production, it may be partially correct, and, of course, the documents could contain several events instead of at most one per document.

This can be complicated by the fact that just as the analyst may add information that was not in the original document, the GTD contains event information that is not included in the summary, and thus could not possibly have been extracted. We try to avoid penalising the AI for this.

Numerous methods have been proposed for measuring similarity [START_REF] Pandit | A comparative study on distance measuring approaches for clustering[END_REF]; [START_REF] Tversky | Studies of similarity[END_REF]; [START_REF] Cohen | Very Fast Similarity Queries on Semi-Structured Data from the Web[END_REF]; [START_REF] Dutkiewicz | Knowledge-based highly-specialized terrorist event extraction[END_REF] to give but a few), but we wanted a measure taking into account the events' four dimensions, and capable of differentiating between different types of event, therefore we put weights a, b, c and d on the dimensions and define:

Overall similarity between two events

σ(E 1 , E 2 ) = aσ C (C 1 , C 2 ) + bσ G (G 1 , G 2 ) + cσ T (T 1 , T 2 ) + dσ A (A 1 , A 2 ) (a + b + c + d) (7.1)
Semantic similarity Because we mapped the values of the WebLab semantic dimension onto those of the GTD (see Table 7.4), we can define the semantic similarity σ C (C 1 , C 2 ) to be 1 if there is a common element between the semantic dimensions of E 1 and E 2 , and 0 otherwise. For example, for C 1 = {Bombing} and C 2 = {Attack, Bombing}, we obtain

C 1 ∩ C 2 = {Bombing}, so σ C (C 1 , C 2 ) is 1.

Geographical (spatial) similarity

We define the geographical similarity

σ G (G 1 , G 2 ) in the same way, that is σ G (G 1 , G 2
) is 1 if there is a common element between the geographical dimensions of E 1 and E 2 , and 0 otherwise. For example, for

G 1 = {Caen} and G 2 = {Rouen} G 1 ∩ G 2 = ∅, and so σ G (G 1 , G 2 ) is 0 Temporal similarity The temporal similarity σ T (T 1 , T 2 ) is 1 for T 1 ∩T 2 = ∅.
Otherwise, if there is no common element, we use partial and derived information. For example, if {October 1969, 07-10, 10-07, 1969-10, 1969, October} October is a common element with T 3 , so we give σ T (T 1 , T 3 ) a value of 1 10 . Finally, we can calculate that 7 th October 1969 was a Tuesday, so σ T (T 1 , T 4 ) = 1 7 . Note that the values 1 10 and 1 7 were chosen by experiment, and highlight the difficulty of giving an exact numerical value to a comparison.

T 1 = {7 October 1969}, T 2 = {7/10/69}, T 3 = {October} and T 4 = {Tuesday}. We can see that T 1 is the same as T 2 , just in a different format, so σ T (T 1 , T 2 ) is 1. Deriving possible partial dates from T 1 = {7 October 1969} gives
Agentive similarity For the agentive similarity, we initially tried the Jaccard similarity given by the number of elements common to each set divided by the total number of distinct elements:

Jacc(X, Y ) = |X ∩ Y | |X ∪ Y |
For example, if the first set A 1 is {Al-Qaeda, Army of Islam} and the second A 2 is {Al-Qaeda, The Foundation}, then

Jacc(A 1 , A 2 ) = |Al-Qaeda| |Al-Qaeda, Army of Islam, The Foundation| = 1 3
The problem is that this does not take into account spelling differences which are very common, for example in the transcription of Arabic names, and both the analysts' corrections and the GTD standardise the spellings. If A 2 contained {Al-Qaida} instead of {Al-Qaeda}, the Jaccard similarity would give

Jacc(A 1 , A 2 ) = |∅| |Al-Qaeda, Army of Islam, Al-Qaida, The Foundation| = 0 4
We therefore decided to use the Levenshtein distance (the minimum number of characters to delete, insert or replace to convert one string into the other) [START_REF] Levenshtein | Binary codes capable of correcting deletions, insertions and reversals[END_REF] on each pair of agents a 1 , a 2 in A 1 , A 2 .

Levenshtein(Al-Qaeda, Al-Qaida) = 1 replacement But, names are often more complex, for example, if we compare Jharkhand anti-Maoist Maobadi Protirodh Committee member with Jharkhand Party member, the standard Levenstein distance is 36, and yet, logically, they are the same entity. Also, the named entity (NE) extractors such as Gate can make partial matches, or over-match. We therefore make this Levenshtein distance "fuzzy" (F L(a 1 , a 2 )) by comparing the substrings [START_REF] Ginstrom | National Consortium for the Study of Terrorism and Responses to Terrorism (START)[END_REF]. This means that we can ignore the prefixes and suffices which are often the parts which are mismatched, whilst still accommodating slight differences in spelling.

CHAPTER 7. BIMBO: A FLEXIBLE PLATFORM

We define the agentive similarity σ

A (A 1 , A 2 ) as σ A (A 1 , A 2 ) = 1 - min(F L(a 1 , a 2 )) min(|a 1 |, |a 2 |) | ∀a 1 ∈ A 1 , ∀a 2 ∈ A 2
if over a certain threshold θ, and 0 otherwise (in practise, θ = 0.45 gave the best results). For example, if A 1 = {Dr Dolittle PhD} and A 2 = {Doolittle}, the standard Levenshtein distance is 7, but the fuzzy Levenshtein distance is only 1, and so the similarity

σ A (A 1 , A 2 ) is σ A (A 1 , A 2 ) = 1 - F L(✟ ✟ ❍ ❍ Dr Dolittle✘ ✘ ✘ ❳ ❳ ❳ PhD, Do ✁ ❆ olittle) |Doolittle| = 1 - 1 9 = 0.889
Taking the problematic example above: if A 1 = {Jharkhand anti-Maoist Maobadi Protirodh Committee member} and A 2 = {Jharkhand Party member}, then the fuzzy Levenshtein calculation gives a distance of 11, and thus a similarity of 1 -11 22 = 0.5. Given two lists of agents, we do a pairwise matching, and use the maximum similarity.

We do not try here to associate named entities such as London / capital of England, but the modularity of the system would allow this.

Example 6 (Full comparison) If we take the two events described in Figure 7.7, we can see that:

• There is an overlap ( Death Event) in the semantic description of the event, and so the semantic similarity is 1.

• The spatial similarity is 1, due to the common element West Bengal.

• We have to rely on the deduction of the day of the week, getting a temporal similarity of 0.143.

• Finally, the agentive similarity, as we have already seen, is 0.5.

If we put equal weights on the dimensions (a = b = c = d = 1), we have an overall similarity of σ(WebLab Event, GTD Event) = 1.0 + 1.0 + 0.143 + 0.5 4 = 0.66075 between the extracted event, and the corresponding event in the GTD. Definition 7 (Quality) The quality of the events extracted (if they exist) weighs this similarity against the time taken to treat the document. More precisely, if the extraction of Êi from document d i took time t with target event E i , we define the quality Q of the extraction to be

MEASURING THE QUALITY OF THE RESULTS

69

Q = σ( Êi , E i )/t if σ( Êi , E i ) > 0 -t otherwise)
Obviously, if there is no event extracted (or the extracted event has null similarity with the target event), then Q = -t.

We thus formalise the fact that the correct extraction of events is primordial, and must be done in a reasonable time, and that the AI should rapidly detect if there is no interesting event to extract.

As we said in section 7.4, the corpus was constructed from the GTD summaries such that the chain should only extract one event from each document. In our earliest tests, the AI learnt to "cheat", by calling the extraction service many times over to extract the same event. Therefore, now, if the AI extracts several events for one target event, we measure the quality of each, and then take their mean. [START_REF] Cheng | Learning similarity functions from qualitative feedback[END_REF] note that manually specifying a similarity measure is extremely hard, and yet the performance of the system often depends on it. We are conscious that none of the above measures are perfect, even if they do allow the AI to learn effectively (see chapter 8). Our objective was always therefore to move away from giving a numerical reward based on this similarity measure to giving a genuinely qualitative feedback (see chapter 9).

Chapter 8

Experiments with numerical rewards

Framework

In our first experiments with numerical rewards, we use standard Q-learning [START_REF] Watkins | Learning From Delayed Rewards[END_REF] as presented in chapter 5. Nonetheless, our contribution includes modelling the improvement of a treatment chain as an RL problem, and in practise, any RL algorithm could be used.

In these experiments, the AI receives numerical rewards as defined in section 7.6 on the quality Q of its results. In production, this corresponds to rewarding it based on the similarity between the event which it has extracted (if any), and the extraction as corrected by the human analyst, considered to be the "perfect" event which could be extracted from the document (if any). This is non-intrusive, as it relies on corrections which the analyst makes as part of his daily routine, but requires fine-tuning of the definition of the similarity and its parameters, a cognitively difficult task, as we've already seen. Note that the numerical reward given to the AI here is the same as that used for assessing the quality of the policy it has learnt, as is standard in RL but not natural in real life. In chapter 10, we remove these requirements and report the results with qualitative rewards.

In our tests, unless otherwise specified, the AI starts learning "from scratch" with no a priori knowledge of the documents or the users needs. We say that such an AI is "untrained". Obviously, this means that we cannot expect good extractions from the start. Rather, we are interested in how fast a "good" policy is learnt (after seeing how many documents), and how good this policy is. Once the AI has treated a certain number of documents, it will have learnt what it considers to be an optimal policy, and we say that it is "trained".

Obviously, here we are testing the learning capacity of an initially untrained AI, but in production we would capitalise on existing expertise by initialising BIMBO with a policy based on that of an existing production chain. With the Q-Learning algorithm, for example, this can be done quite 72 CHAPTER 8. EXPERIMENTS WITH NUMERICAL REWARDS simply by importing the Q-values already learnt. We would launch the expert chain on known documents, and force the algorithm to always choose the "expert" action. The Q-values thus created can be easily stored, and then re-read to initialise the AI.

We performed three types of experiment with numerical rewards, each time comparing the performance of the AI(s) against our reference, the expert chain (as a reminder, the expert chain is a production chain constructed by hand to be optimal for this type of document).

• Firstly, in section 8.2: Trained vs untrained vs expert -we train an AI, and then pit it against an untrained AI and the expert chain on an unknown set of documents to see how effective the training was. We also emulate an analyst's interest in a certain type of event.

• Secondly in section 8.3: Few extractable events -to test the robustness of the learning, we increase the action space, and give an untrained AI a set of documents that contain a very low percentage of extractable events.

• Finally, in section 8.4: Sporadic rewards -to emulate the non-availability of the analyst, we test three untrained AIs with increasingly sporadic rewards. We observe and explain a strange behaviour, and experiment with a different way of reducing the exploration rate ǫ.

Trained vs untrained vs expert

In this section, we first train an AI repeatedly on a small set of 100 documents. We then test the quality of the policy that it learnt during this training on a larger, randomly chosen set of 1000 documents. As a comparison, we also run an untrained AI and the expert chain on the same 1000 documents.

As a reminder, the similarity calculation is

σ(E 1 , E 2 ) = aσ C (C 1 , C 2 ) + bσ G (G 1 , G 2 ) + cσ T (T 1 , T 2 ) + dσ A (A 1 , A 2 ) (a + b + c + d)
For both the training and the tests, when measuring the similarity between the extracted event and that of the oracle, we emphasise the semantic similarity by setting the semantic weight a to 20 and the other dimensions' weights b, c and d to 1. The aim was to reinforce the extraction of event types that the analyst finds interesting (defined here as Bombing), and hence

TRAINED VS UNTRAINED VS EXPERT

73

influence the choice of gazetteer. Note that the heavily weighted semantic similarity means that AI still receives a reward for an event which is not interesting, but it is much smaller than that of an interesting event.

To illustrate this, let us assume that the events that the analyst is interested in are of type Bombing and that we set a to 20 and b, c and d to 1. If the geographical, temporal and agentive dimensions match the expected event perfectly, then

σ G (G 1 , G 2 ), σ T (T 1 , T 2 ) and σ A (A 1 , A 2 ) are all 1.
If the event extracted is of type Bombing, then the semantic similarity is also 1. This means that the similarity between events E 1 and E 2 :

σ(E 1 , E 2 ) = 20 × 1 + 1 × 1 + 1 × 1 + 1 × 1 (20 + 1 + 1 + 1) = 23 23 = 1
If, on the other hand, the event extracted is not of type Bombing, then the semantic similarity is 0. This means that the similarity between events E 1 and E 2 :

σ(E 1 , E 2 ) = 20 × 0 + 1 × 1 + 1 × 1 + 1 × 1 (0 + 1 + 1 + 1) = 3 23 ≃ 0.13 .

Training the AI

We took a set of 100 GTD documents, 64 of which described the "interesting" Bombing events, 17 Injure events, and 19 of which contained no extractable events. We passed these documents through BIMBO 30 times in total. Note: in hindsight we could probably have trained the AI on far fewer documents, but we were young, foolish, and pessimistic. As we said in chapter 7, the actions we made available to the AI were to choose a service from {Tika, NGramJ, GATE, Geo}, to choose a GATE gazetteer, or to STOP the treatment and return any extracted events.

We took three pairs of gazetteers, each pair consisting of a list of verbs and a list of nouns. This gave the AI six possible gazetteer choices. Two of these pairs, the bombing (respectively injure) verbs and nouns contained words likely to trigger the detection and extraction of a event of type Bombing (respectively Injure) were copied from the production gazetteers. The third pair consisted of dummy lists and contained verbs and nouns not present in the GTD. We hoped the AI would take into account the analyst's preference by favouring the bombing gazetteers over the injury gazetteers, and that it would learn to ignore the dummy lists as they never lead to an extraction.

CHAPTER 8. EXPERIMENTS WITH NUMERICAL REWARDS

We wanted to decrease the exploration and increase the exploitation with time. This is because although the exploration can sometimes be useful, especially at the start, to find better treatment combinations, usually it results in partial, faulty or missed extractions which we cannot allow to continue indefinitely. We therefore started with an exploration rate, ǫ of 0.4, then every 5 × 100 documents, we divided this by 2 until we reached 0.1.

Recall the Q-Learning update formula for the Q-values that we gave in Algorithm 1 in chapter 5 on page 39:

Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )(r t+1 + γ max b { Qt (s t+1 , b)} -Qt (s t , a t ))
The value α serves as a modifier on the latest reward gained, whether it be positive or negative. As the documents were heterogeneous, some containing extractable events, and others not, we set the learning rate α to be 0.2. This means that convergence to the exact value function cannot be guaranteed as the rewards will never be given in their entirety to the AI, but it does dampen the negative effect of large variations, for example, if the several documents in a row do not contain extractable events, resulting a long stretch of negative rewards.

Testing the trained AI

In order to test the policy learnt by the trained AI, we set its exploration rate ǫ and its learning rate α to zero. This meant that it followed the learnt policy with no exploration (that it, it always chooses the optimal action), and never updated the Q-values, as the update formula for the Q-values becomes:

Qt+1 (s t , a t ) ← Qt (s t , a t ) + 0 × (r t+1 + γ max b { Qt (s t+1 , b)} -Qt (s t , a t ))
or in other words:

Qt+1 (s t , a t ) ← Qt (s t , a t )
To test the untrained AI, we set the learning rate α to be 0.2, and the exploration rate ǫ to be 0.4 as for the training, but we divided ǫ by 2 every 100 documents until it reached 0.1.

We gave both AIs the choice of all six gazetteers (the bombing, injure and dummy nouns and verbs). The expert chain was given only the bombing gazetteers, meaning that it could extract only the Bombing events. This meant that the expert chain was optimised to extract the events defined as being of interest to the analyst.

We fed BIMBO 1000 documents chosen randomly from the GTD that the AIs "saw" for the first time. Only 29% of these contained "interesting" Bombing events, 7% Injure events, and 64% no extractable events.

As a reminder of the quality measure that we detailed in Definition 7, page 69, BIMBO gave the AI a positive reward for an extraction (unless it was completely dissimilar to the oracle), and a negative reward based on the time it took if it extracted nothing.

The results for the untrained AI (starting from "scratch"), the trained AI and the expert chain are shown in Figure 8.1.

We can see that the untrained AI, after a shaky start, managed to start consistently extracting events after 220 documents. This shows that it generalises well, its performance improving the more different documents it encountered.

Comparing the trained AI and the expert, we can see that the trained AI demonstrated an expert-like policy, extracting 100% of the possible events. This shows that it also generalises well, applying a learnt policy successfully to unknown documents. The only problem was when there was nothing to extract, it did not stop the chain rapidly enough, which is why the negative rewards are lower for the trained AI than for the expert chain.

We also note that the AI learnt to order the chain, for example in the initial state: language " " nounGazetteer none format " " verbGazetteer none interesting false nbServices 0-5 any false timeTakenSoFar 0 the best action learnt was to pass the document to the service Tika to extract the document metadata and to convert it to an XML format which is essential for the following services to function.

In the state corresponding to a document where at least one event is extracted, whatever the values of the other attributes: language en nounGazetteer any format text/plain verbGazetteer any interesting any nbServices any any true timeTakenSoFar any the AI stopped the treatment of that document. It had learnt that once any event had been extracted, even if it was not "interesting", it was not going to do better. Indeed, with one target event per document summary, and the services that we offered it, this was the case. Even if it passed several times through the extractor, it could only extract the same event again and again. We could imagine that if we were to add a translation service, and extractors of different languages, then the AI might be able to wring more information out of the document by doing a first extraction in one language, then translating and doing another in a different language.

As hoped, the AI also learnt to optimise the chain. It did not call the service Geo, and only used the bombing verb list.

As we remarked in the example decision process in section 7.2, it also made an unexpected choice of gazetteer, finally preferring injure nouns to those of bombing. On investigation, we found that the GATE service provided erroneously only uses the verbs for event extraction, ignoring the nouns. This shows that the AI was able to discover strategies which were not clear even to the expert calibrating the chain, and could be an unexpected advantage of our approach. In development, for example, by running the AI as we have here on a set of test documents whose events are already known, we could test new services with two objectives. Firstly, we can show that they improve the extractions, and secondly, we can ensure that they do not exhibit any unwanted behaviour. Unexpected choices by the AI point directly to the part of the service concerned (here the choice of the noun gazetteer for GATE), and could be a useful debugging tool.

In production, this test could also be carried out on a regular basis using the current pool of services. If a service exhibits a surprising behaviour, an alert could be raised with the development team, and another "safe" service substituted automatically whilst the erratic one is investigated.

Few extractable events

We saw in Figure 8.1 that the untrained AI was able to learn to extract events from a set of documents, only 36% of which contained extractable events. In this section, we push this percentage even lower, and test an untrained AI on 5000 unknown randomly chosen documents, only 22.56% of which contain extractable events. Figure 8.2 shows the percentage of extractable events averaged for readability over groups of 100 documents, and we can see that it is not at all evenly spread. For example, the average number of extractable events in documents 1701-1800 drops as low as 7%, but rises to 46% just afterwards in documents 1801-1900.

We removed the semantic bias, setting the event dimension weights a, b, c and d to all be 1, but we kept the interesting event type as Bombing. To further handicap the AI, we also increased the action space by adding two more gazetteers, a pair of event specific gazetteers shooting (nouns and verbs), and a pair of all-purpose gazetteers mixed (nouns and verbs) which contain longer lists of words likely to result in the detection of several types of events. We sought to verify that the AI learnt to use the mixed gazetteers rather than the unique event gazetteers, that it still avoided the dummy gazetteers, and that it was not sensitive to a larger action space.

So why should the AI prefer the mixed gazetteers to the bombing gazetteers if the interesting event type was still Bombing? In fact, the mixed gazetteer allows all the Bombing events plus all the other types of event to be extracted. All extractions get rewarded, and this reflects the fact that the analyst is more likely to prefer to see too many extracted events, rather than risk missing one because it has been misclassified. This means that the AI now has the choice of 15 possible actions -the services {Tika, NGramJ, GATE, Geo}, STOP, plus ten gazetteers (bombing, shooting, injure, mixed and dummy nouns and verbs).

Once again, we set the learning rate α to be 0.2, and the exploration rate ǫ to be 0.4 initially, divided ǫ by 2 every 100 documents. This time, however, we reduced the minimum value of ǫ to 0.05. As we explained for the first set of tests, we want to reduce the exploration over time so that the AI can start choosing the optimal actions more and more frequently. These tests were much longer than the previous set, and we wanted the AI to stabilise In parallel with the learning, every 100 documents, we initialised an AI with the policy learnt at that point. We set the values of ǫ and α to 0 (recall that this means no exploration, and no learning), and tested this policy on 10 unknown documents, all of which contained extractable events. The results are shown in Figure 8.3, and our first observation is that the AI improves, even if the learnt policies are not consistently capable of extracting 100% of the events.

I'm now going to ask you to juggle three different graphs (sorry) so that we can analyse them together. Firstly the percentage of extractable events by groups of 100 documents (Figure 8.2). Secondly Figure 8.3 showing the quality of the policy learnt at the end of each group of 100 documents. Thirdly Figure 8.4 which shows the rewards received by the AI compared to those of the expert chain (again by groups of 100 documents).

In Figure 8.4, we can see that the first extraction did not happen until the document number 173, and that the few extractions that the AI manages from the first 600 documents are of lower quality than those of the expert chain. This ties in with what we observe in Figure 8.3 -the learnt policies are not capable of extracting more than 1 out of 10 of the events. Between documents 601 to 700 (where Figure 8.2 shows us that 39% of the documents contain extractable events), the quality of the AI's extractions seems to improve to equal that of the expert chain, and Figure 8.3 confirms this -the policy learnt after treating document 700 can extract 7 out of 10 events.

Around document 750 (where 31% of the documents contain extractable events), the extraction quality becomes poor again, and between documents 800 to 900 (where 20% of the documents contain extractable events), the AI starts to treat almost all the documents as rapidly as possible suggesting a risk averse behaviour. It prefers to receive a reward of 0 for not doing anything to the document, rather than receiving a negative reward by trying unsuccessfully to extract an event which does not exist. Again, this is confirmed by testing the learnt policy and seeing no extractions.

Between documents 900 to 1300 (where 19%, 14%, 8% and 17% respectively of the documents contain extractable events), we can see that the percentage of events extracted is very low, and that the AI has problems both with the quality and quantity of its extractions. We can see, however, that the quality of the learnt policy is actually increasing.

The AI starts extracting events again between documents 1301 to 1700 (where 37%, 27%, 23% and 26% respectively of the documents contain extractable events) showing that the AI can still learn with relatively few rewards. Checking the learnt policies confirms this impression.

Documents 1701 to 1800 are interesting. Only 7% contain extractable events, and yet the AI still manages to extract just over half of the events. Maybe once the AI has learnt a good policy, it can keep a certain inertia even during a period of famine or maybe it was just lucky. Checking the learnt policy, it seems that it was not luck; after treating document 1800, the AI is capable of extracting 9 out of 10 events with the policy that it has learnt so far.

SPORADIC REWARDS

83

The extractions from documents 1801 to 2300 (where 46%, 25%, 25%, 26% and 28% respectively of the documents contain extractable events) seemed to benefit from the peak of 46% in extractable events, as after a shaky start, the quality and quantity of the extractions seems quite consistently good. However, the learnt policy tells a different story. After documents 1900, 2000 and 2100, the AI cannot extract a single event. This seems counter-intuitive until we remember the very low rate of extractable events just before. The learnt policy has obviously been damaged by the low rate of 7%, and any extractions are due to luck until after document 2200 when we see that the AI has at last managed to learn an expert-like policy, extracting 10 out of 10 possible events.

Between documents 2301 to 2400 (where 18% of the documents contain extractable events) the AI starts well, but then fails to extract most of the events. This is echoed by the learnt policy, which degrades from 7 out of 10 to zero extractions.

We can conclude that the AIs results are certainly affected by the percentage of extractable events, but that it seems to only need around 20-25% of the documents to contain extractable events to learn a good to expert-like policy.

Sporadic rewards

The previous sets of experiments aimed to assess the ability of the AI to extract events from unknown documents, not all of which contained extractable events. In this section, we tested the AI on 5000 documents again, but this time they all contained extractable events. Our aim was to see how quickly the AI could learn an expert-like policy (able to extract all events) given sporadic rewards. We gave the AI the choice of the services or STOP plus ten gazetteers as in section 8.3. In the graphs below, we only show the results for the documents for which the AI exploited, i.e. followed its best policy for the whole treatment (which is why the red expert line varies between plots). We also smooth the curve for readability, taking averages on sets of 100 documents in the same order as they are treated.

We performed three tests:

• First a baseline test in which BIMBO gave rewards after each document, just as we've done so far. This led us to discover a strange behaviour if the AI is allowed too much exploration, and caused us to rethink our ǫ reduction strategy.

• Naturally the analyst is not available 24×7 to consult each document as it is treated. We therefore ran a similar experiment, but BIMBO gave rewards on all extractions only once every 100 documents, hence preventing the AI from learning during those 100 documents.

• Finally, the analysts are not dedicated trainers, and will not correct all extractions. BIMBO therefore only gave a reward at the end of each 100 documents with a probability of 10% on each extraction (otherwise the AI received no reward at all for that extraction).

Initially, Q-learning was run with parameters similar to those of the first experiments: α = 0.2 and ǫ = 0.4, divided by 2 every 100 documents until ǫ = 0.1. In running the first baseline test, however, we noted that although the agent learnt an expert-like policy very quickly, sometimes the performance degraded after a little while (see Figure 8.7) because the AI was choosing to STOP rather than treat the document. On examining the logs, we saw that the AI successfully learns a path s 1 to s n (see Figure 8.8) where s n results in an extraction.

Then if the exploration rate is kept too high, in the state s j , the explore actions gradually lower the Q-value of the pair (s i , a i ) until the action a i is a worse option than STOP (which gives a reward of 0 if done rapidly). The probability of not exploring between s j and s n is very low, as the chain of actions is potentially very long. For example, the time taken to change a gazetteer is negligible, and the AI might choose to do it several hundred times in one treatment. [START_REF] Dahl | Learning While Exploring: Bridging the Gaps in the Eligibility Traces[END_REF] problem with exploring. They claim that "the feedback from the exploring actions will be biased and push the evaluations away from the correct ones".

To confirm our suspicions, we increased the exploration even more, by only dividing ǫ by 2 every 200 documents. In Figure 8.9, we can see that this does indeed have a detrimental effect. We therefore tried a more intelligent reduction strategy for epsilon. Sutton and Barto (1998, page 186, Example 6.6) state that if ǫ is gradually reduced, then Q-Learning will asymptotically converge to the optimal policy. Our problem was therefore finding out which "gradually" worked in our case.

We used

ǫ(s) = 1 (1 + n(s)) β
where ǫ(s) is the epsilon value to use in state s and n(s) is the number of visits to state s so far. To guarantee convergence, we must choose β such that 0.5 < β 1.0 as Sutton and Barto (1998, page 53, Equation 2.8) show CHAPTER 8. EXPERIMENTS WITH NUMERICAL REWARDS the conditions required to assure convergence with probability 1 are:

∞ k ǫ = ∞ and ∞ k ǫ 2 < ∞ and we know that ∞ k 1 n r converges if r > 1 and diverges if r 1
We first tried the most aggressive value of β = 1.0 (see Figure 8.10a), then β = 0.75 (see Figure 8.10b), but found that these cut short the exploration too soon, and the AI never learnt an expert-like policy. We then set β = 0.55 with good results (see Figure 8.11). As a reminder, for this test, BIMBO gave the AI a reward for each document. After 400 documents, the average quality over 100 documents was good, and by the time the AI had treated 800 documents, it was able to extract 100% of the events. More importantly, the extraction quality with this policy was on a par with that of the expert chain. We also note that the ability to extract events and the ability to extract correct events increase together, as the lower quality (compared to the expert chain) that we observe in the first part of plot was due to the processing time, and not the similarity with the target event.

As we said, the analyst is not available to consult each document as it is treated. During the test where BIMBO gave a delayed reward only once 100 documents had been treated (Figure 8.12), the AI was understandably slightly slower to learn, but after 700 documents, it had learnt to extract 100% of the events, and as you can see, the extraction quality was very good. The final test emulated the fact that analysts cannot correct all the documents. BIMBO therefore only gave a reward at the end of a group of 100 documents with a probability of 10% on each extraction (otherwise the AI received no reward at all for that document). Even with such delayed, sporadic rewards, the AI managed to extract 100% of the events after 1200 documents (see Figure 8.13).

Incidentally, you may be wondering why, if all the documents contain extractable events, you can see that both the AI and expert lines dip to zero around document 2500 in Figure 8.10b, 400-600 in Figure 8.12, and five times in Figure 8. [START_REF]<p r o p e r t y name=" morphDilateType " v a l u e="MORPH_RECT" /> <p r o p e r t y name=" k e r n e l S i z e " v a l u e=" 0 , 3[END_REF]. Recall that we chose to only show the results for treatments where no sub-optimal action was chosen (i.e. the AI did not explore). For these documents it explored at least once during their treatment.

Summary

In this section, we performed three types of test in which we gave the AI numeric rewards, which were also used to measure the quality of its results. We compared how well untrained and trained AIs performed against an expert chain and showed that the AI consistently performs well, even with very sporadic feedback, providing that at least 20 -25% of the documents contain extractable events. We also showed that the AI is capable of generalising on unknown documents. The AI's unexpected choice of gazetteer to parametrise the GATE service led us to discover a previously undetected error in that service.

Finally, we observed that the AI sometimes exhibited a strange behaviour, appearing to "forget" what it had learnt. On investigation, we found that the exploration rate remained too high, and so we carried out some tests with a more sophisticated exploration reduction method.

Chapter 9 RL with intuitive feedback

Non-numerical rewards

Traditionally, reinforcement learning algorithms use numerical rewards. This has the advantage of being precise, allowing the agent to learn exactly which policies are the best, and indeed, as we showed in chapter 8, our approach works very well with this type of user feedback. The trouble is, most humans are not comfortable giving precise numeric values. Imagine that you greet your neighbour "How are you this morning?". He is highly unlikely to reply "I'm 7.5 today, 3.2 better than yesterday". He will probably respond "I'm fine thanks, much better than yesterday".

In the same way, when we ask the analyst to define his requirements, it is highly improbable that he can specify a precise numerical value for a similarity such that "an edit on a character costs 0.3", "a character deletion costs 0.6", "a missing first name of length 7 costs 10.5" etc. Or for the treatment time, "15.2 seconds costs 7.9", and so on. Not only is it nigh on impossible to be exhaustive in the list of possible outcomes, but it is also extremely difficult to evaluate each possibility quantitatively.

Our objective was therefore to find a way of learning from qualitative feedback, which the analyst could define easily and intuitively. Weng et al. (2013) point out that the definition of a numerical reward function is non-intuitive, especially when this reward does not represent a physical measure. They treat this problem by offering the agent an ordinal reward from a categorical, completely ordered scale. [START_REF] Weng | Interactive value iteration for markov decision processes with unknown rewards[END_REF] treat a similar problem of non-numeric rewards which can be ordered by a tutor. Both reformulate the Ordinal Reward MDP (ORMDP) [START_REF] Weng | Markov Decision Processes with Ordinal Rewards: Reference Point-Based Preferences[END_REF] as a Vector Reward MDP (VMDP). Similarly, in Gilbert et al. (2015a), the user is asked to provide comparisons, expressed as value vectors, rather than give a specific numerical reward.

To illustrate this idea of value vectors: As a reminder, a policy π is a set of pairs {(s, a)}, i.e. in a given state s, a is the (best) action to perform. An MDP models the probability (as observed by the agent) for each state, for all actions from that state, of arriving in s ′ 90 CHAPTER 9. RL WITH INTUITIVE FEEDBACK and getting reward r.

Instead of using numerical values for the rewards, they are given a "label", such as w i (for example, "an extraction in 10 seconds", or "no extraction"), which is associated automatically with a final state.

Following π then gives a vector whose indices are the probabilities of receiving each reward, e.g.

(p 1 , p 2 , p 3 , p 4 )     w 1 w 2 w 3 w 4    
means that the agent receives the first reward w 1 with probability p 1 , reward w 2 with probability p 2 , and so on.

For [START_REF] Weng | Interactive value iteration for markov decision processes with unknown rewards[END_REF], these strictly ordinal rewards, whilst more natural than a numerical user feedback, can lead to non-intuitive questions for the user : do you prefer 2w 1 + 7w 3 + 3w 4 to 3w 1 + w 2 + 2w 5 ? Those comparisons can be complicated and non-intuitive to evaluate. For instance, could the analyst judge if it is better to have to change one letter of a name, delete a place, add a year and have an extraction two seconds faster than to have to add a name, change the month, but have an extraction one second slower? Using the technique of [START_REF] Weng | Interactive value iteration for markov decision processes with unknown rewards[END_REF] would also mean potentially asking the user questions during the treatment of the document (at the calculation of the policy), which does not fit in with their normal work-flow.

We also considered using the more relaxed landmarks of qualitative reasoning (see [START_REF] Travé-Massuyès | Mathematical foundations of qualitative reasoning[END_REF] for an introduction). This gives us the comparisons +, -, 0, ? (better than, worse than, similar to, or incomparable with the previous). Comparing "nothing extracted" with "a perfect extraction", we could assume -, for instance, or if something was extracted, we would rely on the user to tell us if it were + or 0. This rather Orwellian approach of "double plus good" [START_REF] Orwell | Tandem Library[END_REF] still requires the user to make a non-intuitive cognitive judgement (where would they draw the line between + and 0, for example), and to be consistent in those judgements.

Humans are not very good at giving precise numerical values, or evaluating complex vectors, but they are usually excellent at making simple pairwise comparisons, so we turned to preference based reinforcement learning (PRBL) [START_REF] Busa-Fekete | Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm[END_REF][START_REF] Akrour | APRIL: Active preference learning-based reinforcement learning[END_REF][START_REF] Fürnkranz | Preference-based reinforcement learning: a formal framework and a policy iteration algorithm[END_REF][START_REF] Wilson | A bayesian approach for policy learning from trajectory preference queries[END_REF]Wirth and Fürnkranz, 2013a,b;[START_REF] Wirth | Model-free preferencebased reinforcement learning[END_REF]. This is the integration of two sub-fields of machine learning, namely preference learning and reinforcement learning.

NON-NUMERICAL REWARDS

91

For example, in [START_REF] Fürnkranz | Preference-based reinforcement learning: a formal framework and a policy iteration algorithm[END_REF], they ask the user for their preferences over simulated roll-outs or trajectories. In Figure 9.1, we see that from a given common state s 1 , the agent simulates the trajectories formed by taking each possible action from that state (the "roll-outs"), and then following a given policy until the final states. Maximizing the expectancy of cumulated rewards cannot be done directly as the numerical values of those rewards are not available, but the outcomes can be given a preference order, for instance that τ 1 is preferred to τ 2 , and that τ 2 is preferred to τ 3 . Knowing this preference order, we can then infer a preference order over the actions from the common state s 1 : that a 1 is preferred to b 1 , which is preferred to c 1 . This means that we can infer that Q(s 1 , a 1 ) is greater than

Q(s 1 , b 1 ) which is in turn greater than Q(s 1 , c 1 ). s 0 s 1 s 2 s ′ 2 s ′′ 2 s n τ 1 s ′ n τ 2 s ′′ n τ 3 a 0 a 1 b 1 c 1 Figure 9
.1 -Roll-outs are carried out from s 1 , giving a preference order: τ 1 , τ 2 , τ 3 over the trajectories, and hence over the actions from s 1 .

In our case, as we saw in chapter 6, we define our states as vectors of values, one of which is the detected language. We could therefore imagine that in Figure 9.1, s 1 could be a state where the language detected is Afrikaans. The action a 1 could be a direct extraction from the original Afrikaans, resulting in final state s n . However, [START_REF] Matthew | Using technology recycling to develop a named entity recogniser for afrikaans[END_REF] states that the extraction of named entities from Afrikaans has been neglected in favour of languages such as Dutch. Our South African friends tell us that Afrikaans is similar to Dutch, so we offer an action b 1 which is to translate from Afrikaans to Dutch, and then to use the Dutch extraction rules to finish in state s ′ n . We also know that our richest named entity extraction rules are for the English language, and so we try a third action c 1 , to translate the Afrikaans into English, and then extract, presenting the results in state s ′′ n . We then ask the user to compare the three results, and from their preferences we can infer 92 CHAPTER 9. RL WITH INTUITIVE FEEDBACK which action is the best in state s 1 . This is expensive in terms of processing and user time, however, as the same document would be treated three times, and the comparison between the results would still not necessarily be intuitive, nor would it enter into the analyst's normal daily routine. Also, in attempting to construct a consistent reward function from the preference orders, we risk introducing preferential information which was not given explicitly by the user, therefore giving the agent an unintended bias. For example, the user may prefer τ 1 to τ 2 until he finds out that τ 1 was produced by sheer luck, and that normally the "a 1 " path would produce τ 4 , which is the worst possible result.

SSB model-based approaches

An approach which adapts the preference order decision criteria to ordinal information is probabilistic dominance [START_REF] Busa-Fekete | Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm[END_REF]. Probabilistic dominance aims to maximize the probability of yielding a preferred outcome. More formally, let π 1 and π 2 be two policies and let F 1 , F 2 be two random variables on the final states (f 1 , . . . , f k ) (in our case, the states where the AI chooses the action STOP, or we force it to stop) where P(F i = f j ) is given by the probability of π i achieving result f j , and F x F y indicates that the result achieved by π x is preferred or equal to that achieved by π y . Then according to the probabilistic dominance criterion:

π 1 π 2 ⇔ P(F 1 F 2 ) ≥ P(F 2 F 1 )
In words, policy π 1 is preferred to π 2 if the probability of π 1 doing better than π 2 is greater than the converse.

Probabilistic dominance is actually just a specific type of Skew Symmetric Bilinear (SSB) utility function [START_REF] Fishburn | SSB utility theory: an economic perspective[END_REF]. Given φ, an SSB utility function, φ(π, π ′ ) > 0 means that the user prefers π to π ′ (and conversely φ(π, π ′ ) < 0 means that he or she prefers π ′ to π). If they are similar or incomparable from the user's point of view, then φ(π, π ′ ) = 0. In probabilistic dominance, we restrict the values of φ to 1, 0, or -1, which means that the user need only provide a purely ordinal feedback, without having to specify the degree of preference. [START_REF] Gilbert | Solving MDPs with Skew Symmetric Bilinear Utility Functions[END_REF] showed that it is possible to learn a (potentially mixed) optimal policy with only qualitative feedback, using SSB utility functions to compare policies in MDPs. This frees us from the restriction of totally ordered rewards, allowing the reward function to be expressed in a completely natural manner. The user can indicate preferences over final states, rather than having to give numerical or ranked rewards.

We entered into collaboration with Hugo Gilbert (LIP6, Université Pierre et Marie Curie, Paris) to expand on [START_REF] Gilbert | Solving MDPs with Skew Symmetric Bilinear Utility Functions[END_REF] with respect to our industrial challenge. 1Suppose that the preferences of the decision maker are represented by an SSB function ϕ applied on probability distributions over the set of final states F (the set of possible feedback). Let s 0 be a fixed initial state (in our context, this would be as an unknown document arrives for treatment, i.e. with no information yet extracted, no services called, zero seconds on the clock, etc.), π, π ′ be two policies and p π|s 0 (f ), p π ′ |s 0 (f ) be the probability distributions induced by π and π ′ over final states given s 0 . In other words, p π|s 0 (f ) is the probability that the execution of policy π from initial state s 0 finishes in final state f . The preferences of the decision maker are defined as follows:

π π ′ ⇔ ϕ(p π|s 0 (f ), p π ′ |s 0 (f )) ≥ 0 where ϕ(p π|s 0 (f ), p π ′ |s 0 (f )) = s,s ′ ∈F P(f = s|π, s 0 )ϕ(s, s ′ )P(f = s ′ |π ′ , s 0 ) = t (p π|s 0 (f ))Φp π ′ |s 0 (f )
where Φ[i, j] = ϕ(s i , s j ).

Example 8 To take a concrete example, in Figure 9.2, we see two policies. In the first, in state s 1 , if we take the best action, we have a 10% chance of going to s 2 , and a 90% chance of going to s 3 . From s 2 , we can arrive in states s 4 , s 5 and s 6 . Both s 4 and s 6 offer the same reward w 1 .

If we currently use policy π 1 (Figure 9.2a) and we find the second policy π 2 (Figure 9.2b), how can we tell if π 2 is better than π 1 (is π 2 ≻ π 1 )? Imagine that the users have told us that the final state w 3 is preferred to w 2 , which is in turn preferred to w 1 , that is ϕ(w 3 , w 2 ) = 1 and ϕ(w 2 , w 1 ) = 1. Obviously ϕ(w i , w i ) = 0 as the same result can be neither better nor worse than itself. Note again that we never have to ask for a numerical value of w i , just whether w i is preferred to w j or not. First we work out the vector of probabilities that we receive each reward (accumulated wealth), w i , for each policy:

for π 1 : p π 1 w = (p(w 1 | π 1 ), p(w 2 | π 1 ), p(w 3 | π 1 )) = (0.1 × (0.5 + 0.3), 0.1 × 0.2, 0.9 × 1.0) = (0.08, 0.02, 0.9) for π 2 : p π 2 w = (p(w 1 | π 2 ), p(w 2 | π 2 ), p(w 3 | π 2 )) = (0.5, 0.5, 0)
We then compare the value of each policy by doing a pairwise comparison of the wealth (reward) vectors, and multiplying by the function ϕ(w, w ′ ) which tells us if w is preferable to w ′ :

To compare π 2 with π 1 , we work out ϕ(π 1 , π 2 ) = w,w ′ p(w | π 1 )p(w ′ | π 2 )ϕ(w, w ′ ) = (0.08 × .5) × ϕ(w 1 , w 1 ) + (0.08 × .5) × ϕ(w 1 , w 2 ) + (0.08 × 0) × ϕ(w 1 , w 3 ) + (0.02 × .5) × ϕ(w 2 , w 1 ) + (0.02 × .5) × ϕ(w 2 , w 2 ) + (0.02 × 0) × ϕ(w 2 , w 3 ) + (0.09 × .5) × ϕ(w 3 , w 1 ) + (0.09 × .5) × ϕ(w 3 , w 2 ) + (0.09 × 0) × ϕ(w 3 , w 3 ) = (0.04) × ϕ(w 1 , w 1 ) + (0.04) × ϕ(w 1 , w 2 ) + (0) × ϕ(w 1 , w 3 ) + (0.01) × ϕ(w 2 , w 1 ) + (0.01) × ϕ(w 2 , w 2 ) + (0) × ϕ(w 2 , w 3 ) + (0.045) × ϕ(w 3 , w 1 ) + (0.045) × ϕ(w 3 , w 2 ) + (0) × ϕ(w 3 , w 3 ) = 0.04 × 0 + 0.04 × (-1) + 0 × (-1) + 0.01 × 1 + 0.01 × 0 + 0 × (-1) + 0.045 × 1 + 0.045 × 1 + 0 × 0
And so, the comparison between the two policies, ϕ(π 1 , π 2 ) = 0 -0.04 -0 + 0.01 + 0 -0 + 0.045 + 0.045 + 0 = 0.06

In other words, ϕ(π 1 , π 2 ) > 0, so π 1 is preferred to π 2 (and intuitively, we can see that this is the case).

To apply this to our treatment chain we need to define a preference order over the final states. Each final state has candidate options described on k user-defined criteria (c 1 , . . . , c k ). For example, each final state could be a vector, and the criteria could be:

• extraction time t • event dimension corrected • event confirmed correct • event deleted • event not consulted • etc.
This could then give final state vectors: If these criteria are ranked by order of priority c 1 ≥ c 2 ≥ . . . ≥ c k , then the preferences between two candidate options o 1 and o 2 are:

o 1 o 2 ⇔c 1 (o 1 ) > c 1 (o 2 ) or c 1 (o 1 ) = c 1 (o 2 ) and c 2 (o 1 ) > c 2 (o 2 ) or c 1 (o 1 ) = c 1 (o 2 ) and c 2 (o 1 ) = c 2 (o 2 ) and c 3 (o 1 ) > c 3 (o 2 )
or . . . If the partial preference relations are as defined in Figure 9.3, for instance, then "confirmed correct" is preferred to "edited", but the user would refrain from comparing "deleted" to "not consulted" or to "fast extraction".

Then φ(

→ v 1 , → v
2 ) > 0 if they are equal on all dimensions but one, and

→ v 1 > → v 2 on this dimension.
Hugo suggested two model-based algorithms which could be adapted to an SSB type reward: R-Max [START_REF] Brafman | R-max-a general polynomial time algorithm for near-optimal reinforcement learning[END_REF] and V-Max [START_REF] Rao | V-MAX: A General Polynomial Time Algorithm for Probably Approximately Correct Reinforcement Learning[END_REF], which learn the underlying MDP instead of a policy directly. R-Max builds a model of the environment in which it finds itself, initialising the rewards for unknown parts to a maximum to encourage exploration. It then uses the model to make decisions, and updates the rewards to their observed values after trying a given transition m times. In this way, once a transition is considered "learnt", the agent goes off to explore other areas which still offer the attractive maximum reward.

SSB MODEL-BASED APPROACHES

V-Max is similar to R-Max, but instead of updating the transitions only after trying one transaction m times, they are updated after each visit. In this way, V-Max uses the information it acquires a lot more efficiently.

We implemented both algorithms with numeric rewards initially, which we hoped would be a stepping stone to a qualitative (SSB) implementation, and compared them to the Q-learning algorithm that we saw in chapter 8.

For the tests, similarly to subsection 8.2.1, 30 iterations (batches) of 100 documents were passed through BIMBO to train the AIs. This time, however, the dimension weightings were equal (a = b = c = d = 1).

We can see the results in Figure 9.4. Each figure shows the quality of results of the 30 th iteration as a blue dotted line, compared with the expert chain as a red solid line, and includes the explorations.

As expected, the Q-Learning algorithm performs very well (see Figure 9.4a), and the results (for example, towards the end) that are sometimes lower than those of the expert were due to exploration.

The R-Max results were very disappointing (see Figure 9.4b). It explores a lot, and its theoretical guarantees of convergence unfortunately do not translate into a very good practical performance. It tends to explore in depth a tiny sub-space of the space / action couples (s, a) before passing to the next sub-space. In other words, it goes round in circles, before going elsewhere, then it turns in circles in that "elsewhere" before going somewhere else, etc. The good news was that it tended towards rewards of 0 from below which implies an improvement nonetheless, i.e. it did not learn to extract the events, but at least it was quick to extract nothing.

We hoped that V-Max would perform better, but it never even managed to get off the ground, taking a long time to not extract anything (see Figure 9.4c).

Another problem in our particular case with both V-Max and R-Max (and we suppose any other model-based approach), was that the length of the paths followed by the agent rendered the calculation time of the policy (the Bellman backup) prohibitively long. We set a time limit for the treatment, but changing a gazetteer is practically instantaneous, and so the agent may change it several hundreds of times in a few seconds, potentially taking it to a different state each time.

SSB Q-Learning -a model-free approach

To tackle the problem of a model-based SSB approach not being viable in our case, we proposed a model-free approach based on the SSB function and Q-Learning. [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF] details a new algorithm, SSB Q-learning (see Algorithm 2). We invite you to read the original article for more details, but we give an overview of the general SSB Q-Learning algorithm here. Our formalisation of the user feedback for our application is detailed in chapter 10.

When we ask the analyst to define his requirements in the simplest terms, they are likely to reply something like "I prefer a false extraction to a missed extraction, as I don't want to lose information, and I'd rather it were as fast as possible". Note that there are two preferences expressed here, which although they impact each other (one might suppose that speed could result in poor extraction quality), cannot naturally be linked numerically. We therefore want to reward the agent based on the decoupled user preferences: "I prefer an extraction to no extraction" and "I prefer it to be as fast as possible". We've already seen that this preferential information can be expressed as a partial order over possible results achieved (f 1 , . . . , f k ) (seen as final states) given by the human agent. For instance f i can stand for "a good extraction in 10 seconds" or "no extraction in 5 seconds", etc.

Like Q-learning, SSB Q-learning uses epsilon greedy (EG) exploration, and it updates the Q-values similarly. However, instead of having the numerical values of the rewards given by the environment, they are defined by φ and the past experiences of the agent i.e. the frequencies p t with which it has achieved each possible final result so far (the resulting numerical value is denoted by R pt (s t+1 )). Intuitively, the algorithm continuously tries to act better, according to φ, than it has so far (Algorithm 2).

Algorithm 2: SSB Q-learning [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF] Data: MDP M, SSB function ϕ 1 while True do 2 Choose a t using the EG exploration strategy 3 Play a t , observe s t+1 , and let

r t+1 = R pt (s t+1 ) 4 Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )(r t+1 + max b { Qt (s t+1 , b)} -Qt (s t , a t )) 5 if s t+1 is a final state f i and exploration is off then 6 p t+1 = p t + β p (1 i -p t ) 7 # β p is 1 η+1 . 8
# η is the number of times p has been updated.

Example 9 We will expand Example 7 of the Rowett dice game given in the article [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF] here to aid understanding. The game consists of three non-standard dice A, B and C, and the aim is to throw the highest total score. The player first chooses to throw the die A (action a A ) or not (action a BC ). In the latter case, he then must choose to throw either die B (action a B ) or die C (action a C ). The dice are six-sided, but the sides are not numbered 1 to 6. Instead, die A has one 1 and five 4's, die B has five 3's and one 6, and die C has three 2's and three 5's. Incidentally, these dice have the interesting property of being non-transitive, which means that A will usually beat B, B will usually beat C, and C will usually beat A. The probabilities of throwing each number for each die are shown in Table 9.1.

Table 9.1 -The Rowett dice probabilities.

1 2 3 4 5 6 p A 1/6 0 0 5/6 0 0 p B 0 0 5/6 0 0 1/6 p C 0 1/2 0 0 1/2 0 Figure 9
.5 shows the current position. We're halfway through a game at time t, and the Q-values learnt so far are:

Q(s 0 , a A ) = 0.1; Q(s 0 , a BC ) = 0.2; Q(s BC , a B ) = 0.3; Q(s BC , a C ) = -0.7
The probability vector p t = ( 1 5 , 0, 1 5 , 1 5 , 1 5 , 1 5 ) shows that the agent thinks that there is a 1 in 5 chance of throwing a 1, 3, 4, 5 or 6, but zero chance of 100 CHAPTER 9. RL WITH INTUITIVE FEEDBACK throwing a 2. The dice have been thrown n = 80 times in total, 20 times die A (episode of length 1), and 30 times either B or C (episode of length 2). This means that a final state has been reached 50 times, so η p = 50, and β p = 1/(η p + 1) = 1/51 ≃ 0.02. The state-action pair (s 0 , a BC ) has been updated 30 times so η s 0 ,a BC = 30 and α t (s 0 , a BC ) = 1/31 2/3 ≃ 0.1. Similarly, η s BC ,a B = 20 and α t (s BC , a B ) = 1/21 2/3 ≃ 0.13

s 0 s BC s A s B s C a BC a A a B a C Figure 9
.5 -The player first chooses not to throw die A, then chooses to throw die B.

From state s 0 , the player follows the policy, and as Q(s 0 , a A ) < Q(s 0 , a BC ) has chosen not to throw die A. They are now in state s BC and the Q-value Q(s 0 , a BC ) is updated as in Algorithm 2. That is:

Q(s 0 , a BC ) ← Q(s 0 , a BC ) + α t (s 0 , a BC ) × (R pt (s BC ) + max b { Q(s BC , b)} -Q(s 0 , a BC )) Because we're not yet in a final state, R pt (s BC ) = 0, so Q(s 0 , a BC ) ≃ 0.2 + 0.1 × (0 + 0.3 -0.2) = 0.21
Again following the policy, the player will now throw the die B, and

Q(s BC , a B ) ← Q(s BC , a B ) + α t (s BC , a B ) × (R pt (s B ) + max b { Q(s B , b)} -Q(s BC , a B )) So Q(s BC , a B ) ≃ 0.3 + 0.13 × (R pt (s B ) + 0 -0.3)
Because we're in a final state, R pt (s B ) = 1 T i Φp t . The aim is to roll high, so a roll of 3 (r 3 ) on the die will beat a 1 (r 1 ), that is, φ(r 3 , r 1 ) = 1, but a 6 (r 6 ) would beat the 3. This means that if we roll a 3, then i = 3 and

R pt (s B ) = 1 T 3 Φp t = (0, 0, 1, 0, 0, 0)            r 1 r 2 r 3 r 4 r 5 r 6 r 1 0 -1 -1 -1 -1 -1 r 2 1 0 -1 -1 -1 -1 r 3 1 1 0 -1 -1 -1 r 4 1 1 1 0 -1 -1 r 5 1 1 1 1 0 -1 r 6 1 1 1 1 1 0                    1/5 0 1/5 1/5 1/5 1/5         = 1 5 - 1 5 - 1 5 - 1 5 = - 2 5 Therefore Q(s BC , a B ) ≃ 0.3 + 0.13 × (-0.4 -0.3) ≃ 0.2
We update the probabilities: This approach, as you will see in chapter 10 works very well. The restriction imposed that the rewards can only be given in the final states obviously is not a problem for our use-case, and even works to our advantage, because we only want feedback on the whole treatment, i.e. the final state representing the results.

p t+1 = p t + β t (1 i -p n ) = ( 1 5 , 0, 1 5 , 1 5 , 1 5 , 1 5 ) + 1 51 × (1 3 -( 1 5 , 0, 1 5 , 1 5 , 1 5 , 1 5 )) = ( 1 5 , 0, 1 5 , 1 5 , 1 5 , 1 5 ) + 1 51 × (- 1 5 , 0, 4 5 , - 1 5 , - 1 5 , - 1 5 )

Chapter 10

Experiments with intuitive feedback

Framework

In chapter 8, we carried out a proof of concept of our approach. We modelled the document treatment chain as an MDP, and we saw that the AI successfully learnt how to construct an efficient chain using reinforcement learning with numerical rewards.

Here, we change the experimental context and test algorithms which are only given qualitative, non-numerical rewards against a baseline which receives numerical rewards.

As we've already stated in section 9.3, to cater for qualitative, intuitive user feedback, which standard approaches cannot handle, we use SSB Q-learning [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF]. Because this algorithm was hot off the press, we wanted to put it through its paces. We tried it with a variety of parameters to test its robustness, comparing it with a standard Q-Learning with the same parameters.

We saw in section 2.5 that [START_REF] Knox | Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and performance[END_REF] suggest that where the agent should be task-focused, the discount value γ should be high. We therefore chose to use two values for γ: 0.9 and 1.0.

We've also seen (for example in section 8.4), that the amount of exploration that the agent is allowed to make has a big impact on the success of its learning. We tried two reduction strategies for the exploration rate, or EG parameter ǫ. A slow reduction (divided by 2 after 2500 documents) and a faster one (divided by 2 after 1000 documents) down to a minimum in both cases of 0.05.

With both SSB Q-Learning and Q-Learning, we used eligibility traces. We go into a little more detail of what these are in chapter 12, but recommend that you read Sutton and Barto (1998, Chapter 7) for an excellent explanation. When using eligibility traces, a decay parameter λ ∈ [0, 1] controls how far new experiences are back-propagated along the trace of decisions taken thus far. At one extreme, λ = 0 corresponds to the algorithms 104 CHAPTER 10. EXPERIMENTS WITH INTUITIVE FEEDBACK as described in Algorithm 1 and Algorithm 2, i.e. with no back-propagation along the trace. At the other extreme, λ = 1 corresponds to Monte-Carlolike RL algorithms with full back-propagation. We wanted to find a value that was not too small, not too big, but just right [START_REF] Southey | The story of the three bears[END_REF], and so we tried three values for λ: 0, 0.95 and 1.0.

For each setting of these parameters, we compared the performance of the expert chain and those of three AIs learning "from scratch":

• QL: Q-learning(λ) with numerical feedback, as we used in chapter 8, but with eligibility traces;

• MAG: SSB Q-learning(λ) as presented in chapter 9, expressing preferences of different MAGnitudes: φ ∈ {-1000, -100, -10, 0, 10, 100, 1000} (see subsection 10.1.1 for more details).

• DOM: SSB Q-learning(λ) as presented in chapter 9, with probabilistic DOMinance, that is, purely ordinal feedback: φ ∈ {-1, 0, 1} (see subsection 10.1.2 for more details);

To summarise, this gave us 36 combinations of tests:

• Discount factor γ: 0.9 and 1.0.

• EG parameter ǫ divided by 2 after 2500 documents or 1000 documents.

• Decay parameter λ: 0, 0.95, 1.0.

• Three AIs: QL, MAG and DOM;

To measure the performance of all approaches, we ran them, initially untrained, on a set of 5 000 GTD documents (presented in the same order, and only seen once in all experiments) from which the expert chain can extract events.

The available actions were given (as in section 8.3) by choices from ten gazetteers, the services Tika, NGramJ, GATE, Geo, and STOP.

We measured the quality of the treatment of each document as described in section 7.6, and emphasise that even though the quality of the extraction was measured in the same way for all approaches (including the expert chain), the feedback given for the AIs to learn from was different.

Both Q-learning(λ) and SSB Q-learning(λ) were run with α set as in [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF], i.e. decreasing as the number of visits to the current state / action pair grows.

FRAMEWORK 105

Recall that SSB Q-learning only requires feedback on the relative quality of two extractions. This means that the user no longer has to specify numeric rewards for the AI. Instead, he specifies his preferences in a simple, intuitive way, and we interpret those preferences as final states (representing the treatments finishing on those states).

MAG definition

The first type of SSB feedback, MAG, is a hybrid between a numeric and a purely qualitative feedback. For the MAG feedback, we emphasised the importance of extracting events ("I prefer an extraction to no extraction"), compared to the importance of extracting the exact target events ("I prefer a correct extraction to an incorrect one"), in turn compared to the importance of efficiency ("I prefer it to be as fast as possible").

Intuitively, it is therefore a middle ground between QL and DOM, and can be seen as a weighted form of probabilistic dominance. Yet such feedback remains quite natural. We formalised the MAG feedback given to the AI as follows:

• A final state f is hugely preferred under MAG to final state f ′ , i.e.

φ MAG (f, f ′ ) = 1000 if and only if:

f extracted an event and f ′ did not.

• A final state f is largely preferred under MAG to final state f ′ , i.e. φ MAG (f, f ′ ) = 100 if and only if:

both f and f ′ extracted an event but f extracted an event of higher quality (as measured in section 7.6) than f ′ .

• A final state f is slightly preferred under MAG to final state f ′ , i.e. φ MAG (f, f ′ ) = 10 if and only if:

both f and f ′ extracted an event of similar quality or neither extracted an event, but f was faster than f ′ by margin seconds (where margin is parametrisable).

• A final state f is considered equivalent under MAG to final state f ′ , i.e. φ MAG (f, f ′ ) = 0 if and only if:

f is similar to, or incomparable with f ′ with respect to the previous conditions.

Note: as we initially had no idea how the algorithm SSB Q-Learning(λ) would behave, the values 10, 100 and 1000 were chosen rather arbitrarily. In section 10.4, we explore the effect of varying these magnitudes, and see that although they are not as difficult to refine as the definition of similarity, they still require a small effort to tune correctly.

DOM definition

For DOM, we took the simplest user preferences "I prefer an extraction to no extraction" and "I prefer it to be as fast as possible" to test just how little information we could give the learner. We therefore formalised the feedback given to DOM in our experiments as follows:

• A final state f is preferred under DOM to final state f ′ , i.e. f ≻ DOM f ′ ⇔ φ DOM (f, f ′ ) = 1 if

and only if:

f extracted an event and f ′ did not, or neither treatment or both treatments resulted in an extraction, but f was faster than f ′ by margin seconds.

• A final state f is considered equivalent under DOM to final state f ′ , i.e. f ∼ DOM f ′ ⇔ φ DOM (f, f ′ ) = 0 if and only if:

f is similar to, or incomparable with f ′ with respect to the previous conditions.

We thus encouraged the AI, in an arguably very natural manner, to extract events first, and to do so fast, or to recognise quickly that there are no events to extract. This relies on the correlation, demonstrated in chapter 8, between the ability of the treatment chain to extract any event, and its ability to extract the correct event.

As we stated just above, we took the simplest user preferences for DOM, but obviously, this preference relation could be completed with additional information over the obtained results, such as the perceived quality of the extraction, etc.

In these tests, we set margin to be 5 seconds which is between 1 2 and 1 4 of a typical treatment time (this obviously depends on the speed of the machine used to run the tests, and margin should be adjusted accordingly).

RESULTS

107

Results

Our intuition was that QL should be more effective than MAG, and MAG more effective than DOM, as QL receives precise numeric rewards which reflect the evaluation criteria, giving it much more information than MAG, and MAG receives more than DOM as it takes into account the quality of the extraction. However, we demonstrate in this section that MAG and DOM are perfectly realistic approaches in an industrial setting, and may even be preferable to QL.

The results are shown in Figure 10.1-Figure 10.4. On each plot, we show the extraction quality of a particular feedback approach (blue dotted line) against that of the expert chain (red solid line). We only consider the results for the documents for which the AI "exploited", i.e., followed its "best" policy for the whole treatment (which is why the red line varies between plots). For readability, we smooth the curve, taking averages on sets of 50 documents in the same order as they are treated.

RESULTS

111

Table 10.1 gives a cross-reference of the varying parameters and the resulting graphs with a subjective colour code indicating how good to excellent (light to dark green) or bad to awful (light to dark red) the results were. The first column shows the rate (in documents) at which ǫ is reduced, the second the values of γ tested, the third the values of λ. The fourth column shows the results for Q-Learning(λ) and the last two, the results for SSB Q-Learning(λ) with the definition of φ used.

QL results

Glancing at Table 10.1, it is evident that QL is split down the middle with 50% of good to very good results, and two rather inconclusive results for γ = 1 and λ = 1 and 0.95.

Examining the more detailed graphs in Figure 10.1-Figure 10.4, we can see that QL can indeed give excellent results, but that it is quite sensitive to changing parameters.

• For both ǫ reduction strategies, the combinations of γ = 0.9 and λ = 0 (Figure 10.1a and Figure 10.3a) or λ = 0.95 (Figure 10.1g and Figure 10.3g) gave excellent results. For example in Figure 10.1g the AI learnt a "good enough" policy after only 250 documents, and a expertlike one after 1200 documents.

• Testing γ = 0.9 with the faster ǫ reduction strategy (Figure 10.3a, Figure 10.3d and Figure 10.3g) gave excellent results whatever the value of λ.

• With γ = 0.9 and λ = 1 and the slower ǫ reduction strategy (Figure 10.1d), however, the results were mediocre. The AI started to extract events around the 2000 document mark, but then after around 4100 documents chose to STOP rather than treat the document. This is likely to be a similar problem to the one we noted in section 8.4, where the explorations were eroding the Q-Value of a key state / action pair.

•

MAG results

Table 10.1 shows us that MAG's results are consistently good to excellent. Specifically, from Figure 10.1-Figure 10.4, we can see that the AI with MAG feedback appeared very robust to changes in parameters, quickly learning a "good enough" policy in every case. That is, it learnt more or less rapidly to extract the events, even if it was not always as rapid as the expert chain. In fact, in section 10.4, we show that although it is not sensitive to the changes in γ = and λ, it is sensitive the scale of feedback offered.

• Setting γ = 0.9 and λ = 1.0 for the faster ǫ reduction strategy (Figure 10.3e) also allowed the AI to learn an expert-like policy.

• The combination γ = 0.9, λ = 0.95 for the slower ǫ reduction strategy (Figure 10.1h) is particularly good, learning a "good enough" policy almost straight away, and improving rapidly by speeding up to achieve an expert-like policy after 3 000 documents.

SUMMARY

113

• When λ = 1, there is a back-propagation of the rewards along all of the trace (although it may be cut at a certain length depending on the implementation) and the AI has no need to revisit the state / action pairs on the trace to feel the effect of the rewards at state s 0 . This means that the exploration can be decreased faster. For λ = 0.95, however, this back-propagation is shorter, which means that the AI needs to revisit the pairs along the trace more times before the effect of the feedback reaches state s 0 . This explains why the slower ǫ reduction strategy was necessary in this case.

DOM results

We can see in Table 10.1 that DOM's results are usually good to excellent, but that when they are bad, they are very, very bad. Indeed, in Figure 10.1-Figure 10.4, we can see that DOM was fairly sensitive to the choice of γ and λ (but still not quite as much as QL was).

• With γ = 0.9 and the slower ǫ reduction strategy the results were awful for both λ = 1 and λ = 0.95 (Figure 10.1f and Figure 10.1i).

Similarly for γ = 0.9 and λ = 1 and the faster ǫ reduction strategy (Figure 10.3f). The AI never managed to extract even a single event.

• However, reasonably good results were achieved with γ = 0.9, λ = 0 for the slower ǫ reduction strategy (Figure 10.1c), and with γ = 0.9, λ = 0 and γ = 0.9, λ = 0.95 for the faster strategy (Figure 10.3c and Figure 10.3i).

• With γ = 1, the results were good to excellent regardless of the ǫ reduction strategy (Figure 10.2c, Figure 10.2f, Figure 10.2i, Figure 10.4c, Figure 10.4f and Figure 10.4i). The best results amongst these are for γ = 1, λ = 1 (Figure 10.2f), which learnt a "good enough" policy after 1000 documents and stabilises with an expert-like policy after 3750. Note that we continued this run on a further 5000 documents (not shown), and it stayed expert-like. This confirms the suggestion of [START_REF] Knox | Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and performance[END_REF] that γ should be as high as possible for a task-focused performance.

Summary

We noticed that the AI was capable of improving on a chain which was already good (where the events were extracted correctly but which took a CHAPTER 10. EXPERIMENTS WITH INTUITIVE FEEDBACK few seconds longer than the expert chain). It learnt to speed up, eventually matching the expert chain, for example in Figure 10.3e. This increase in speed is very important, as in the worst cases, it was taking up to 24 hours to run 5000 documents when it should take around half that time.

We also noted that SSB Q-Learning sometimes learnt an expert-like policy but then the performance degrades. For example, in Figure 10.1e and Figure 10.3c, even though the events are still well extracted, it starts taking too long. The logs show that the AI learnt that passing through GATE from a given state gives a good reward, and if γ and λ are not correctly set, although it takes longer, it starts to prefer this action to stopping, and has to be stopped forcibly by BIMBO.

We noted a slight improvement in the results with a faster reduction in ǫ (every 1 000 documents vs every 2 500), that is, with less exploration overall.

QL can give excellent results, but is (very) sensitive to parameter variation and depends on numerical feedback, which is non-intuitive for the user to provide.

MAG seems to offer a good middle ground between a purely numeric and a purely qualitative feedback. It is quite robust to parameter choice (see also section 10.4), uses mostly intuitive feedback, and still learns a good to expert-like policy very quickly.

DOM only requires purely ordinal feedback, and yet with the correct parameters is able to learn expert-like policies, proving to be a very interesting approach for situations where it is impossible to gather a precise numerical feedback.

All in all, the experiments demonstrate that it is feasible to automatically improve a document treatment chain, and even to learn one from scratch, in settings where very little or no numerical information is given.

More MAG results

As we mentioned above, the set of values for φ of -1000, -100, -10, 0, 10, 100 and 1000 were chosen by instinct, and we wondered what would happen if we varied them. We took one of the most successful settings for MAG, that is γ = 0.9, λ = 0.95 and ǫ divided at the slower rate every 2500 documents.

First we tried a linear progression, setting the possible values of φ to be -30, -20, -10, 0, 10, 20 and 30. We ended up with the results shown in Figure 10.5. As you can see, the first try (Figure 10.5a) was a complete failure. In the second try (Figure 10.5b), the AI started extracting the events slightly late, but the results were good. We then tried a linear increase of 100, which left us puzzled, as we expected it to be similar to the linear increase by 10. In Figure 10.6, you can see the results of putting φ in -300, -200, -100, 0, 100, 200 and 300. We tried several times, but either the algorithm was unlucky every time, or this range of φ with these parameters is not suitable for some reason.

MORE MAG RESULTS

115

We then decided to go to the other extreme, and increase the distance between the feedback steps by multiplying by 100 instead of by 10. In Figure 10.7, you can see the results of putting φ in -100 000, -1000, -10, 0, 10, 1000 and 100 000. We were tantalised to see that these results are not great either, the first time, but reasonable the second time.

We wondered if these differences were because we had changed the code since the previous tests were run, so we re-ran the control with φ in -1000, -100, -10, 0, 10, 100 and 1000 (Figure 10.8). We were reassured to get similar results to those in Figure 10.1h on page 107.

We can draw two lessons from this little experiment. The first is that even the best AIs sometimes get bad luck, and for the results to be conclusive, we should have run them several times and taken the average. Unfortunately, each trial takes around 12-24 hours to complete, and we could not carry on the thesis indefinitely, so we had to stop. Secondly, you can see that the tuning of the magnitude for φ is not a completely trivial task after all, even if it is easier than determining an exact numerical value for a similarity. This renders the results of DOM, the purely qualitative solution even more attractive. Chapter 11

CHAPTER 10. EXPERIMENTS WITH INTUITIVE FEEDBACK

Conclusion to Part I

In this, the main part of the manuscript, we detailed how we responded to the challenge laid down by our CIFRE contract. We modelled the IE treatment chain used to extract OSINT events from open-source documents as an MDP, and its improvement as a reinforcement learning problem.

We presented a proof-of-concept in which we used a modular framework, BIMBO, to test untrained AIs in a variety of situations given numerical rewards. We showed that the AIs are capable of constructing a chain step by step, that they generalise well on unknown documents, and that they learn even when given meagre source information, or sporadic feedback.

Encouraged by this success, we wanted to move away from the standard approach of rewarding the agent with quantitative, or numerical feedback. It is non-intuitive for the analyst, and cognitively difficult to define. We therefore investigated various qualitative feedback strategies, and we presented a novel algorithm, SSB Q-Learning [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF].

We applied SSB Q-Learning to our treatment chain, and we showed that even with purely qualitative feedback, we still get excellent results.

In the following Part II, we describe two additional adventures which were born from the main part of the thesis. The first is Dora, a variation on the Q-Learning algorithm which aims to lose less information when exploring. The second is the application of Bimbo to an image analysis project.

117

Chapter 12 Dora -exploiting QL explorations

Introduction

In chapter 10, we said that we were using eligibility traces, but we did not really explain what they were. We go into more detail here, but still recommend that you read Sutton and Barto (1998, Chapter 7). They explain that there are two ways to view a followed policy: the first is "a theoretical forward view", the second "a mechanistic backward view".

In their forward view, at time t, we stand at state s t and see the policy followed stretching out into the future: We update the current state based on what we can see in the state s t+1 , that is, at time t + 1, which is updated based on s t+2 and so on.

s t s t+1 s t+2 s t+n . . .
Unfortunately, not many people possess a time machine to be able to go forward in time to see the value of state s t+n in order to update state s t+n-1 and so on, all the way back to state s t , so Sutton and Barto (1998) introduce a "mechanistic backward" view: In this view, we've already arrived at state s t+1 by following the policy (taking optimal actions in each state). From here, we can see by how much our previous estimates for each state were out, and we can back-propagate the changes back through the states in the path to state s t-n . For example, the expected value of each state might increase if the agent has just received a better reward than it thought it would.

s t-n s t-1 s t s t+1 . . .
In order to keep a record of the path the agent followed (s t-n , . . . , s t-1 , s t , s t+1 ), [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] use an eligibility trace. Each time the agent visits a state, we add it to the trace.

For Q-Learning(λ) (hereafter referred to as QL(λ)) [START_REF] Watkins | Learning From Delayed Rewards[END_REF]; [START_REF] Peng | Incremental multi-step q-learning[END_REF]; [START_REF] Baird | Residual algorithms: Reinforcement learning with function approximation[END_REF]; [START_REF] Cichosz | Truncating temporal differences: On the efficient implementation of TD (lambda) for reinforcement learning[END_REF]; [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]; [START_REF] Wang | Backward Q-learning: The Combination of Sarsa Algorithm and Q-learning[END_REF]; . . . ], the eligibility traces record the stack of state / action pairs enacted during a learning episode. In our case, a learning episode is the treatment of a single document for which the reward is only given at the very end.

To see why we would want to use eligibility traces, recall the Q-Learning algorithm (Algorithm 3):

Algorithm 3: Q-learning Data: MDP M 1 while True do 2
Choose a t in s t using the EG exploration strategy 3 Play a t , observe s t+1 , and let

r t+1 = R(s t+1 ) 4 Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )(r t+1 + γmax b { Qt (s t+1 , b)} -Qt (s t , a t ))
With no eligibility traces, the potential value max b { Qt (s t+1 , b)} of state s t+1 is only back-propagated one time-step to the state / action pair (s t , a t ). Given that in our case, the agent only gets a reward in the final states, and that it may try several hundreds of actions in one episode, it can take a very long time before the effect of these rewards is seen in the initial state. Using eligibility traces, on the other hand, enables the rewards observed to be back-propagated down the stack (or trace), thus speeding up learning by percolating the values back towards the initial state more quickly.

We assumed above that the agent follows the greedy policy (the one which gives the largest expected rewards), but we did not explain what happens if it does not. In fact, [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] state that "we can use subsequent experience only as long the greedy policy is being followed". As soon as the 12.1. INTRODUCTION 123 agent selects an exploratory, non-greedy (sub-optimal) action, the returns observed "no longer have any necessary relationship to the greedy policy".

In standard QL(λ), therefore, after an exploratory action, the eligibility trace is cut (reset to an empty stack), meaning that any good results found further on can take a long time to percolate back to the initial state [START_REF] Dahl | Learning While Exploring: Bridging the Gaps in the Eligibility Traces[END_REF]. In the worst case, where most of the actions are explorations, or where the reward is given only at the end of a (very) long path, the benefits of the eligibility traces can be practically eliminated (Figure 12.3).

. . .

Jackpot

Figure 12.3 -A frustrating exploration gap in a very long chain, just before a large reward, where standard QL(λ) cuts the trace, preventing the backpropagation of the information.

Example 10 Let us look in Figure 12.4 at a small example of how this trace cutting by QL(λ) affects the back-propagation (for simplicity, we focus only on the final back-propagations).

• Figure 12.4a: We see that there are two rewarding states: s 4 which gives a small reward of 10, and s 5 which gives the jackpot of 1000.

• Figure 12.4b: From state s 1 , the agent takes the optimal action a 0 (it exploits) to state s 2 [trace={(s 1 , a 0 )}]. It then exploits a 1 to state s 3 [trace={(s 1 , a 0 ), (s 2 , a 1 )}], and finally exploits (as the two actions a 2 and a 3 apparently have equal value) action a 2 to state s 4 , so now [trace={(s 1 , a 0 ), (s 2 , a 1 ), (s 3 , a 2 )}].

• Figure 12.4c: It receives a reward of 10, which it can back-propagate down the trace to pairs (s 3 , a 2 ), (s 2 , a 1 ) and (s 1 , a 0 ).

• Figure 12.4d: We start a new episode, and the agent as before exploits action a 0 to state s 2 [trace={(s 1 , a 0 )}], and exploits a 1 to state s 3 which gives [trace={(s 1 , a 0 ), (s 2 , a 1 )}], but then it chooses to explore action a 3 to state s 5 . This cuts the trace [trace={}]. s 1 s 2 s 3 s 4 s 5 a 0 : 10 a 1 : 10 a 2 : 10 a 3 : 0 1000 10 (d) A new episode: from state s 1 , the agent exploits action a 0 , then from state s2 exploits action a 1 , but, in state s 3 , it chooses the sub-optimal action a 3 (it explores) which cuts the trace. 12.2. INTRODUCING DORA 125

Introducing Dora

Obviously these exploration gaps are sometimes inappropriate. With the benefit of hindsight, in Example 10, the action a 3 which hit the jackpot actually reflected the optimal policy better than the action a 2 which led to the smaller reward, and so cutting the trace was unnecessary.

We wondered if we could use this hindsight to "exploit the explores", and we came up with Dora, an adaptation of QL(λ), which does just that.

The principle behind Dora is simple. Our aim is to avoid cutting the trace on an explore if possible by testing whether the new experience gained concerning that explore-action now makes it an exploit-action. If it does, then we treat it as if it had actually been an exploit (which in retrospect is the case), and we keep the trace intact enabling the propagation of results from further in the run back across this "explore" join.

We will formalise this in section 12.3, but first:

Example 11 Let us see in Figure 12.5 how Dora would change the situation given in Figure 12.4.

• Figure 12.5a: As before, there are two rewarding states: s 4 which gives a small reward of 10, and s 5 which gives the jackpot of 1000.

• Figure 12.5b: As before, we build up the trace [trace={(s 1 , a 0 ), (s 2 , a 1 ), (s 3 , a 2 )}] by exploiting from states s 1 , s 2 and s 3 .

• Figure 12.5c: As before, the reward of 10 is back-propagated down the trace to pairs (s 3 , a 2 ), (s 2 , a 1 ) and (s 1 , a 0 ).

• Figure 12.5d: We start a new episode. As before, Dora exploits action a 0 to state s 2 [trace={(s 1 , a 0 )}], and exploits a 1 to state s 3 which gives [trace={(s 1 , a 0 ), (s 2 , a 1 )}], BUT when she chooses to explore action a 3 to state s 5 , something different happens. Dora waits before clearing the trace to see what the outcome will be.

• Figure 12.5e: She receives the jackpot of 1000. This means that she now knows that action a 3 in state s 3 gives an expected value of 1000, i.e. Qt+1 (s 3 , a 3 ) = 1000. In other words, the action a 3 that she has just taken is now the best action in state s 3 , i.e. the new value Qt+1 (s 3 , a 3 ) of 1000 beats the old maximum value max b { Qt (s 3 , b)} of 10, so she treats it as if it had actually been an exploit. She keeps the trace intact and adds the new pair [trace={(s 1 , a 0 ), (s 2 , a 1 ), (s 3 , a 3 )}]. The jackpot can now be back-propagated all the way back to the state s 1 . s 1 s 2 s 3 s 4 s 5 a 0 : 10 a 1 : 10 a 2 : 10 a 3 : 0 1000 10 (d) A new episode: from state s 1 , Dora exploits action a 0 , then from state s2 exploits action a 1 . In state s 3 , she chooses the sub-optimal action a 3 (she explores), but waits before cutting the trace.

s
s 1 s 2 s 3 s 4 s 5 a 0 : 1000 a 1 : 1000 a 2 : 10 a 3 : 1000 1000 10 (e) Dora receives the reward of 1000. The best action in state s 3 is now a 3 , which means that the trace can be kept intact, and the jackpot can now be back-propagated all the way to state s 1 .

Figure 12.5 -An example of how Dora keeps the trace intact on an explore which turns out with hindsight to be a "best action", thus allowing the backpropagation of useful learnt values.

12.3. FORMALISATION 127 12.3 Formalisation

12.3.1 Q-Learning(λ)
Before we look at the mechanics behind Dora, let us examine QL(λ). We can see in Algorithm 4 that as in Q-Learning with no eligibility traces (Algorithm 3), the agent chooses an action a t in state s t and observes in which state s t+1 it lands and what reward r t+1 it receives. However, instead of back-propagating just one time-step, as in Q-Learning with no eligibility traces:

Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )(r t+1 + γ max b { Qt (s t+1 , b)} -Qt (s t , a t ))
on Algorithm 4: line 4, we update the Q-Values using Algorithm 5. Data: Given: s t , Qt (s t , a t ) ; Played: a t ; Observed: s t+1 , r t+1

1 δ t ← r t+1 + γmax b { Qt (s t+1 , b)} -Qt (s t , a t ) 2 if a t was argmax b { Qt (s t , b)} then //a
exponent ← 0 2 for (s i , a i ) in reverse(trace) do 3 Qt+1 (s i , a i ) ← Qt (s i , a i ) + λ exponent α t (s i , a i )δ t 4 exponent + +
Let us look at Algorithm 5 in more depth:

• On Algorithm 5: line 1, we calculate the temporal difference δ t to be

r t+1 + γmax b { Qt (s t+1 , b)} -Qt (s t , a t )
This temporal difference is the change over one time-step in the predicted value of Q(s t , a t ), and should look familiar. If we multiply it by the learning rate α t (s t , a t ), and add Qt (s t , a t ), we get Qt+1 (s t , a t ):

Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )δ t
• On Algorithm 5: line 2, we test whether or not the action a t was an optimal action in state s t , that is, whether it was an exploit or an explore.

• If action a t was an exploit, then we add the state / action pair (s t , a t ) to the trace, and back-propagate the temporal difference down through the trace (we will look at the back-propagation below).

• If action a t was an explore, then we clear the trace, and do the one timestep update that we see in the standard Q-Learning without eligibility traces. We then start the trace again by adding the state / action pair (s t , a t ), as if we follow the policy from this point, we can safely back-propagate to here.

Let us look at the back-propagation Algorithm 6 now:

• The trace is treated in reverse order so that the most recently visited states are updated first. This is because a decay factor λ (which is why the algorithm is called QL(λ)) is used to make sure that the states closest to state s t in the trace are more affected by the change than those further away. [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] liken this to shouting the 12.3. FORMALISATION 129 results back down the trace -the closer you are, the louder and clearer you hear the message, but as you get further away, the message gets quieter. This is because the actions are stochastic; the further back you go in the trace, the less likely you would be to come back to exactly the same state, even by following the policy.

• λ is decayed using the exponent, which is initialised to 0, and augmented by 1 for each state / action pair in the trace.

Dora

As we did for QL(λ), we will show you the algorithms behind Dora, and walk you through them step by step. As you can see, Algorithm 7 and Algorithm 4 are identical, and the only change in Algorithm 8 is in the addition of line 7-line 9, which we explain here.

We saw above that Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )δ t

We can test this new Q-value for (s t , a t ) against the previous maximum Qvalue for state s t taken over all possible actions. If it is still less than this maximum, then the action was and remains an explore. If, on the other hand, it is now greater than or equal to the previous maximum, it means that action a t becomes an optimal, or exploit action in state s t . We calculate the temporal difference ∆ t where:

∆ t ← Qt (s t , a t ) + α t (s t , a t )δ t -max b { Qt (s t ,
exponent ← exponent 0 2 for (s i , a i ) in reverse(trace) do 3 Qt+1 (s i , a i ) ← Qt (s i , a i ) + λ exponent α t (s i , a i )δ t 4 exponent + +
There are two important things to note here. The first is that we have not yet added the state / action pair (s t , a t ) to the trace, and the second is that we back-propagate with the exponent of λ initialised to 1 instead of 0. This is because, in examining state s t+1 (the result of taking action a t in state s t ), we have actually used a tiny time-machine to jump two time-12.4. EXPERIMENTS 131 steps with respect to the last element in the trace, the state / action pair (s t-1 , a t-1 ).

We need to imagine that we can go back in time to when we took action a t in state s t . If we had known then what we know now, we would have added (s t , a t ) to the trace and back-propagated the temporal difference ∆ t down the trace. This means that (s t-1 , a t-1 ) would have been the second pair in the trace, and the corresponding exponent would have been 1.

However, the current Q-value Qt (s t , a t ) for the pair (s t , a t ) is not based on our new knowledge, so we cannot use the temporal difference ∆ t to update Qt (s t , a t ). Instead, it needs to be updated with respect to the temporal difference δ t which really applies at time t. We therefore perform a one-step back-propagation using the now familiar update Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )δ t and only add (s t , a t ) to the trace once we have safely parked the timemachine.

You can see that in this way, if a t becomes an optimal action in s t , we gain two advantages. Firstly, we back-propagate the temporal differences δ t and ∆ t . Secondly, we keep the trace intact, enabling the propagation of results from further in the run back across this "explore" join. This is in sharp contrast with QL(λ), which does neither.

Experiments

We tested Dora against QL(λ) taking the average results over at least 50 runs each time. As the paths in the treatment are potentially very long, the discount value γ and the decay rate λ were both set to 0.9. This ensures that a significant proportion of the temporal difference is back-propagated further down the trace than it would have been with a lower value of λ, and that the agent seeks long-term gain rather than immediate rewards. The learning rate α was set to 0.2. To avoid wasting time calculating negligible changes, and the risk of memory explosion, we also limited the trace to a maximum length of 100 pairs.

We tested both with a constant ǫ of 0.2, and a decreasing ǫ(s) as in section 8.4, i.e.

ǫ(s) = 1 (1 + n(s)) β
where ǫ(s) is the epsilon value to use in state s, n(s) is the number of visits to state s so far, and β is 0.55.

CHAPTER 12. DORA -EXPLOITING QL EXPLORATIONS

We measured the quality of their learning by recording the evolution of the distance of their current value function (which measures the best possible value that the agent could achieve from a given state) V t from the optimal V * at each time-step (where s 0 is the starting state):

episodic-distance(V * , V t ) = |V * (s 0 ) -V t (s 0 )|
Note, we also measured the L 2 and infinite distances, but have not shown them as they gave the same curves:

L 2 -distance(V * , V t ) = ( s (V * (s) -V t (s)) 2 ) 1/2 infinite-distance(V * , V t ) = max s {|V * (s) -V t (s)|}

Random MDP tests

We first tested the comparative performance of Dora and QL(λ) on a series of randomly generated MDPs from 10 to 100 states, and 5 to 10 actions. Each state / action pair (s, a) had a randomly chosen, unrestricted number of next states, and were allocated integer rewards generated randomly between 0 and 100.

Figure 12.6a and Figure 12.6b are representative of the results we obtained. We observed that Dora consistently learnt faster than QL(λ), and the larger the MDP, the more significant her advantage. 12.4. EXPERIMENTS 133

Cliff-walk MDP tests

Intuitively, we thought that Dora should perform even better on "long thin" or "cliff-walk" MDPs with long trajectories, and lots of exploration potentially necessary to reach the goal. We constructed such an MDP (see Figure 12.7), where every action except one leads back to the initial state with probability 0.5, or stays on the same state with probability 0.5, obtaining a zero reward. Only one action advances the agent towards the goal with probability 1.0. The reward given for reaching the final state was 100 times the number of states. We ran tests on a series of these MDPs from 5 to 15 states and 2 to 10 actions.

s 0 s 1 s 2 s m a n a 0 , . . . , a n-1
a n a 0 , . . . , a n-1 ; p = 0.5 a 0 , . . . , a n-1 ; p = 0.5 r = 100 × m Figure 12.7 -"Long thin" MDP (n actions, m states) with p(s i , a 1 . . . a n-1 , s 0 ) = 0.5 and p(s i , a 1 . . . a n-1 , s i ) = 0.5 (zero reward); p(s i , a n , s i+1 ) = 1.0. State s m is the only one to offer a reward of 100 × m.

We found that on this particular MDP, Dora consistently significantly outperforms QL(λ) (Figure 12.8a), especially with a decreasing ǫ (Figure 12.8b).

Naive QL(λ)

We thought that an interesting baseline for assessing the efficiency of Dora could be a naive version of QL(λ) with no trace-clearing at all as mentioned at the end of Sutton and Barto (1998, Chapter 7, Section 7.6):

One could imagine yet a third version of QL(λ), let us call it naive QL(λ), that is just like Watkins's QL(λ) except that the traces are not set to zero on exploratory actions . . . We know of no experience with this method, but perhaps it is not as naive as one might at first suppose.

We ran some preliminary experiments which suggest that on long thin MDPs, especially with a decreasing ǫ, there is little difference between Dora and Naive QL(λ) (see Figure 12.9). On generic MDPs, however, Naive QL(λ) gives much worse results than both QL(λ) and Dora (see Figure 12.10). In fact, according to [START_REF] Wyatt | Exploration and inference in learning from reinforcement[END_REF] Naive QL(λ) is not guaranteed to converge.

Experience replay

The idea behind Dora is quite simple and natural, but to the best of our knowledge, it has not been developed like this before. Initially, however, we wondered if the principle of Dora could be argued to resemble that of experience replay [START_REF] Lin | Programming robots using reinforcement learning and teaching[END_REF] which [START_REF] Schaul | Prioritized experience replay[END_REF] define as follows:

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. [START_REF] Lin | Programming robots using reinforcement learning and teaching[END_REF] was motivated to develop experience replay by two perceived inefficiencies in the Q-Learning proposed by [START_REF] Watkins | Learning From Delayed Rewards[END_REF] in his thesis. The first was the waste of the agent's experiences, which are only used once to update the Q-value. [START_REF] Lin | Programming robots using reinforcement learning and teaching[END_REF] argues that some of these experiences are costly to acquire, or rare, and should be used more effectively. The second inefficiency was the order in which the experiences are presented to the agent. As the agent is exploring the world, it experiences the effects of its actions in chronological order. According to [START_REF] Lin | Programming robots using reinforcement learning and teaching[END_REF], this is less effective than presenting the experiences in reverse-chronological order.

In standard experience replay, the experiences (s i , a i , s i+1 , r i+1 ) of the agent are collected and stored for an entire lesson: {(s 0 , a 0 , s 1 , r 1 ), (s 1 , a 1 , s 2 , r 2 ), (s 2 , a 2 , s 3 , r 3 ) . . . (s n , a n , s n+1 , r n+1 )} where s 0 is the initial state, and s n is the final state. Each lesson is chosen either randomly or prioritized (see below) and replayed at the end of one or CHAPTER 12. DORA -EXPLOITING QL EXPLORATIONS more episodes. This enables the agent to learn and self-improve even when a teacher is not available.

In fact, there are several important differences between Dora and experience replay. Firstly, Dora is not model-based, she has fewer parameters to tune (as we're not testing for replays), and consumes less memory (as no storage is required for the lessons), whilst still giving excellent results.

Of course, with experience replay, there is also a danger of introducing statistical bias if the lessons to be replayed are not representative, which is not a problem with Dora. Several methods have been suggested to overcome this weakness by prioritising the lessons selected, for example [START_REF] Schaul | Prioritized experience replay[END_REF], [START_REF] Sander | Experience replay for real-time reinforcement learning control[END_REF], [START_REF] Moore | Prioritized sweeping: Reinforcement learning with less data and less time[END_REF] and [START_REF] Peng | Efficient learning and planning within the dyna framework[END_REF]. For a more complete overview, see Sigaud and Buffet (2010, Chapter 2).

Also, Dora exclusively uses online learning where the updates occur after each state visit, which is more efficient and faster than "offline" learning [START_REF] Wiering | Speeding up Q(λ)-learning[END_REF], which experience replay relies on heavily, where the updates occur only after the end of a trial.

Related and future work

We are obviously not the first to propose improvements to QL(λ), and we give an overview of some of the techniques that we found most interesting below. In many cases, Dora could be combined with them simply by adding the check in Algorithm 8, line 7-line 9 and modifying the back-propagation to take into account the exponent parameter. [START_REF] Wyatt | Exploration and inference in learning from reinforcement[END_REF] gives us some possible ideas for improving the performance of Dora. He compares the effect of various values of α and λ on replacing and accumulating traces. [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] note that if a state is already in the trace, and it is revisited, then performance can be improved by resetting the decay of the trace (replacing), rather than allowing the decay to increase (accumulating). To explore the sensitivity of the trace to exploration, Wyatt (1998) carries out two types of test. During the first, explores are left in the trace (which he calls uncorrected). In the second, he resets the trace on exploration (which he calls corrected). To use his terminology, we therefore used accumulating corrected traces for QL(λ) and Dora, and accumulating uncorrected traces for Naive QL(λ). He shows that accumulating and reducing traces have similar performance, except at λ = 1 when accumulating becomes unstable and replacing continues to improve. He also finds that the corrected trace performs better than the uncorrected 12.6. RELATED AND FUTURE WORK 137 trace, and that the uncorrected trace does not converge, which confirms what we saw in Figure 12.10.

Another interesting idea for improving QL comes from Strehl et al. ( 2006) who introduce Delayed QL, proving that it is PAC (Probably Approximately Correct) and comparing it with R-Max. Their algorithm uses "learn" flags to determine if an update to the pair (s, a) is allowed. After m sample updates of (s, a), if the "learn" flag is true, an update is attempted. This minimises the effect of randomness. If this update fails, then the flag is set to false, and can only be reset to true if another Q-value estimate is updated.

Wiering and Schmidhuber (1998) also try to reduce the number of updates with their algorithm. Fast QL is online, and uses "lazy learning" to only update when needed. They keep track of the changes in a global variable, and then make a local update to the Q-value before an action is selected. After the execution of an action, they perform a global update: update V(s t+1 ) by calling a local update on the next possible state / action pairs, update global variables, then update the current Q-value and trace value and replace the current local value by its global value. Their main advantage over the standard QL(λ) is the reduction in CPU time (but it still takes twice as long as QL with no traces). The learning rate is unchanged.

Like [START_REF] Dahl | Learning While Exploring: Bridging the Gaps in the Eligibility Traces[END_REF], we recognise that exploration can distort the learning. They refer in their Section 2 to the fact that the exploration in standard QL(λ) breaks the trace, and therefore the propagation of feedback to earlier states is delayed. They estimate the cost of not following the greedy policy, i.e. the difference between the exploratory action and the ideal action, and then give these estimates as rewards to compensate the agent for following what it perceives as inferior actions. Because this led to instability, they limit this reward to a maximum value.

Van Hasselt (2010) postulates that QL(λ) is over-optimistic, and to mitigate this, proposes the Double Q-Learning algorithm. Double QL maintains two sets of Q-Values based on the same problem but different experience samples. It then uses one to update the other, and both to choose the next action. He shows (on roulette and grid world experiments) that Double QL converges faster than QL. In [START_REF] Van Hasselt | Deep reinforcement learning with double q-learning[END_REF], they apply Double QL to deep reinforcement learning, giving the results over several Atari games. This gives us another potential avenue to explore -the application of Dora to deep RL, or machine learning. In chapter 13, we use Dora to parametrise two image treatment services, and we would like to expand this to parametrising a deep learning neural network.

In this case, we might imagine combining Dora and experience replay. As Tambet (2016) points out, the Q-function can be approximated using a con-volutional neural network, but an approximation of Q-values using non-linear functions is unstable. He increases the stability when training the network, by randomly giving stored experiences instead of the most recent transition. This helps to keep the network from getting distracted by similarities in successive training samples. [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF] also use experience replay with a neural network, but they select an experience to replay randomly from the stored transition set at every step, rather than sporadically.

It would be interesting to run experiments in a wider variety of settings, for example, with rewards which reduce the optimism rather than increase it, and with several optimal paths in a "long thin" MDP. We might also imagine testing a case similar to the long thin MDP in Figure 12.7, except that the non-advancing actions a 1 to a n-1 actually result in a small reward. This could happen, for example if we want the AI to concentrate on extracting "interesting" events as in section 8.2, when we offer a smaller reward for any extraction deemed not "interesting". In this case, as [START_REF] Moore | Prioritized sweeping: Reinforcement learning with less data and less time[END_REF] found, insufficient exploration would be dangerous as the small reward would tempt the agent away from the longer path to the large reward.

We also conjecture that there are families of "long thin" MDPs for which we can prove that Dora learns exponentially faster than QL(λ) or other algorithms. Apart from the improved algorithm, we believe that such formal results would help gain insight into the interplay between exploration and back-propagation.

Summary

We presented Dora, a variation on Q-Learning(λ), which exploits explores whenever they turn out to be the best action with hindsight. This allows us not only to back-propagate the temporal difference δ t at time t, but also to keep the trace intact, enabling the propagation of results from further in the run back across this "explore" join. Dora therefore learns consistently faster than the standard QL(λ), which does neither.

We gave test results based on two types of MDP -randomly constructed and cliff-walk, and showed that Dora consistently outperforms QL(λ).

We also tried Naive QL(λ), and found that in a cliff-walk MDP it gives similar results to Dora, but that in a random MDP it failed to converge.

Finally, we discussed other methods, including experience replay and how it differs from Dora, and possible future work.

Chapter 13

Application to image analysis

Motivation

One of the business areas of Cordon Electronics DS2i is the maintenance of hardware and systems. Some of these date back decades, and their documentation has been scanned as pdfs and archived. These pdfs cover everything from the technical details of a power supply to the specification of a motherboard, to templates for other documents (see Figure 13.1). Unfortunately, when the documents were scanned, they were only given a number with no indication of their contents. Given that there can be several thousands of documents for one project, searching for the dimensions of a particular obsolete component, for example, can be a long and painstaking task.

Title detection

Valentin Laforge, an internship student from the University of Caen, Normandy, specialising in signal-treatment, was given the objective of automatically detecting the titles of the documents through image analysis. The premise was that any suggestion of a title, even if it was not completely correct was better than nothing, as it would aid in the indexing of the document, and in the search for the contents.

He created two web services, an image pretreatment service and an optical character recognition (OCR) service to identify the most likely titles for a document and present them to the user. We give an overview of the techniques that he implemented below, but invite you to read his internship report "Vision par Ordinateur: Segmentation, Extraction et Reconnaissance Visuelle d'Élements dans des Contextes Variants" ("Computer Vision: Segmentation, Extraction and Visual Recognition of Elements in Various Contexts"), available from the University of Caen, France.

First, the pretreatment service cleaned the image to remove noise (see Figure 13.2). The method used was non-local means denoising [START_REF] Buades | Non-Local Means Denoising[END_REF]. This differs from local means denoising which considers each pixel one by one, and takes the mean colour of the pixels in close proximity. The problem is that the closest pixels along the side of a character on a page, for example, are not at all homogeneous, and their average only adds more noise. In non-local means denoising, the mean is taken over larger areas, which leads to a cleaner, more uniform effect than the localised methods.

TITLE DETECTION 141

Second, as the documents had been scanned by hand, many were slightly lopsided (see Figure 13.3), and so it was necessary to reorient them. The pretreatment service calculated the dominant lines in the document, and then took the average of their perpendiculars to establish the degree of rotation and automatically straighten them. For the most part, the degree of rotation was quite low (between 5 and 20 • ), but sometimes the document was upside down and therefore impossible to rectify correctly.

CHAPTER 13. APPLICATION TO IMAGE ANALYSIS

To finish the image pre-treatment, the image was dilated and eroded. Dilation is a little like blowing up a balloon. It removes the fine details, for instance the hole in the letter "p". Erosion is like peeling a potato -it pares away the edges to remove extrusions. The combination of the dilation and erosion removed the details of the text, and a binary mask was then applied, which enabled the detection of rectangular regions (see Figure 13.4). The second service applied OCR to decode the text in those regions. Certain characteristics tend to define a title. For instance, they might be in a larger font than the rest of the text, bold, italic, or may be outlined in some way. In order to detect and recognise an object in an input image, a fixed size window is used to examine the pixels group by group, and it may be necessary to scale the given input image so that the object fits into the window. A "pyramid" of images in various sizes is therefore created and the window applied to each.

Finally, for each document, the OCR service suggested several titles, with a confidence rating for each (see Figure 13.5).

APPLICATION OF BIMBO

Application of BIMBO

The parameters for the image pre-treatment and OCR were established by expertise and observation of test results, and were set up to cater for the widest possible range of cases. These parameters, however, were insufficient for certain specific cases, for example, the document shown in Figure 13.3b, where the title contains a logo in place of the "A". We therefore decided to apply BIMBO to continuously improve the title suggestions by establishing the best parameters to use for each image on a case by case basis. Guillaume Leroy, internship student from the University of Rouen, Normandy, integrated BIMBO with the two web-services.

Recall the modules of BIMBO (Figure 7.1 on page 52). In Figure 13.6 we show how we adapted her to image detection with only a few changes. Instead of constructing a treatment chain for a document, the AI now has to decide on a parameter configuration. Only when it chooses STOP is the document sent to the image processing services. The "Dynamic Router" has therefore been replaced by a simple definition of the end-point of the first of the services (from which it is passed automatically to the second). The "State / Action Conversion" has obviously changed too (the possible states and actions are detailed in section 13.4). The "Similarity Calculation" has also changed, as detailed in section 13.5. The "Controller" is still responsible for lining up the documents, and handling the stopwatch. The "Reinforcement Learning Algorithms" module remains the same (although we only used Dora), and we still use the "Reporting" module to produce logs etc.. Each choice of value for each parameter was one action, and the combinations of values for each of these parameters defined the states:

BIMBO

• The first part of the state were the document characteristics:

a calculated indicator of the page's complexity, for example, whether the page was typeset, or contains an image such as a company logo;

image dimensions -width and height, which are linked and depend on whether the document is portrait or landscape;

detected language of the document.

• The second part of the state depends on the variable parameters used by the image pretreatment service and changeable by the AI:

the erosion and dilation parameters (which control the amount of detail that is added or removed as in Figure 13.4);

• The third part of the state depends on the variable parameters used by the OCR service and changeable by the AI:

the reduction factors -portrait, landscape, width and height (used to create the pyramid of images);

the number of rectangular selections to analyse;

the kernel parameters (the kernel function is used in the classification of the object detected in the image).

This gives 1 944 000 states and 43 actions in total. Note that this is huge compared to the experiments that we carried out in Part I, and reassures us that the performance of our approach does not seem to be affected by a large state-action space.

Figure 13.7 -The state offered to the AI, consisting of the document characteristics and the variable parameters with their possible values.

< !--The v a r i a b l e p a r a m e t e r s f o r t h e p r e t r e a t e m e n t s e r v i c e --> " morphErodeN " = " 2 " ; " morphDilateN " = " 9 " ; " morphErodeType " = "MORPH_ELLIPSE" ; " morphDilateType " = "MORPH_RECT" ; " k e r n e l S i z e " = " 0 " " w i d t h K e r n e l D i l a t a t i o n R a t i o " = " 1 . 5 " ; " h e i g h t K e r n e l E r o s i o n R a t i o " = " 1 . 3 3 " ; < !--The v a r i a b l e p a r a m e t e r s f o r t h e OCR s e r v i c e --> " p o r t r a i t R e d u c t i o n " = " 2 " ; " l a n d s c a p e R e d u c t i o n " = " 3 " ; " w i d t h R e d u c t i o n F a c t o r " = " 1 . 5 " ; " h e i g h t R e d u c t i o n F a c t o r " = " 2 . 5 " ; " r e d u c t i o n F a c t o r " = " 45 " ; " n u m b e r O f S e l e c t i o n s " = " 6 " ; " a r b i t r a r y Z o o m F o r K e r n e l S i z e " = " 1 5 0 0 0 0 " ; Valentin constructed a corpus of 42 documents, each of which was associated with the title that a user would expect to see in an oracle.

An individual score IS (title) was calculated for each suggested title based on the Fuzzy Levenshtein similarity between it and the oracle FLS (title, oracle) combined with the OCR's confidence C (title) in its detection:

IS (title) = a * FLS (title, oracle) + b * C (title) a + b
In order to take the confidence of the OCR into account whilst still relying mostly on the similarity, a was set to 75, and b to 15.

Tests

First, we ran the corpus of 42 documents with no AI, and obtained a baseline expert score of 0.81, which is shown as the red line in the following graphs.

To calculate the reward, we took the average of the individual scores for all the suggested titles. This gave us an overall score for the document. The AI was rewarded with this overall score as feedback on the parameters chosen for that document treatment.

We ran three tests using Dora, each with an exploration rate ǫ starting at 0.2, and reduced every 84 documents (twice through the corpus):

• During the first tests, this division was by 2.0, and we noted (Figure 13.9) that Dora stabilised very rapidly at 0.85.

• We then divided ǫ by 1.05, allowing Dora to explore for much longer. By the time it had treated the corpus 80 times, it still had not stabilised (Figure 13.10).

• Finally, we took a middle ground (Figure 13.11), dividing ǫ by 1.5, and Dora stabilised at a result of 0.86, beating the expert parameters of 0.81 by 0.05 or 6%. Figure 13.11 -The quality of the image analysis given ǫ starting at 0.2, divided by 1.5 every two times through the corpus, and stabilising at 0.86, beating the expert parameters by 0.05 or 6%.

13.7. SUMMARY 149

Summary

The corpus, although tiny, was constructed based on as many different types of document and title as possible. Dora learnt, not only to parametrise the services correctly, but to do so on a case by case basis, beating the expert parameters convincingly, even on the "problem cases". This validates the flexibility of our approach, showing that it can be easily and successfully applied to other settings than an IE document treatment chain.

Part III

Conclusion 151

Chapter 14

Discussion and perspectives

Conclusion

Summary

Let's go back to our bizarre car factory, and see how they are doing.

They hired a BIMBO to head up their customer feedback department, and she, in turn, hired a team of crack AIs. Each AI was equipped with the blueprints of all possible car models, and put in charge of organising the factory.

The AIs were on performance-related pay, with bonuses for happy customers. To measure the customer satisfaction, they issued a questionnaire every time a car was sold. Some customers didn't bring the questionnaires back immediately, and the vast majority were not even filled in, but the AIs still managed to earn a very good living, and in general customer satisfaction increased.

Then a bright spark at the factory suggested the monitoring device, designed to work alongside the questionnaires. The AIs were delighted, as this meant that they were still paid, even if customers did not want to fill in a questionnaire. Customers were thrilled too. They no longer had to score mint against olive, or asparagus against avocado, they could just say "I like green, preferably dark".

BIMBO noticed that the AIs were sometimes rather slow to file their paperwork, and so she hired a personal assistant, Dora to help them out.

The factory became such a success that they branched out, and bought up a tiny image analysis company.

Implications

Although we applied this approach to an IE document treatment chain, there is no reason why it should not be applied to other processes, from the design of factory floors to route-planning. The ability to mix and match the qualitative and numeric feedback means that the system can be finely tuned to meet the users' requirements.

CHAPTER 14. DISCUSSION AND PERSPECTIVES

The ability to effectively store the knowledge of the users in the "heads" of the AIs means that the longer the system is used, the more it should be able to deal competently with even the most esoteric of situations. The system becomes a knowledge bank, formalising the users' skills and experience. As people leave, or join a company, the system remains the connecting thread between them.

We can also use the unexpected choices that the AI makes, such as its surprising choice of the gazetteers in section 7.2, to perform "sanity checks" on a system. In the case of error in one of the services, it reduces developer labour cost by pinpointing the service and the functional behaviour at fault. In the case where it is not due to a code error, it could point to a misunderstanding of the user requirements.

Limits

Complexity of modelling: One of the limits of our approach is that the modelling task is not a simple one. Firstly, the measurable characteristics of the environment must be identified, along with all their possible values. A decision must then be taken whether to group those values together to avoid a state / action space explosion and speed up the learning, and if so, what size the groups should be, or whether to keep them separate to increase the accuracy of the learning. Secondly, all possible actions that might influence the environment must be listed.

We suspect that it should be possible to mix qualitative model learning [START_REF] Pang | Learning qualitative differential equation models: a survey of algorithms and applications[END_REF] gives a recent overview) with the ability to learn the attributes of a state. For example, [START_REF] Bratko | Learning qualitative models[END_REF] learn qualitative models from the operators control strategy, and [START_REF] Karami | Apprentissage de connaissances d'adaptation à partir des feedbacks des utilisateurs[END_REF] learn which attributes of a state are important. [START_REF] Trampus | Constructing domain templates with concept hierarchy as background knowledge[END_REF] examine the extraction of structured information from natural language text. Given a document, they try to detect which properties are the most relevant to extract, giving as much information, concisely about the document, as possible. Their "properties / slots" are equivalent to our (event) dimensions, and we suspect that this approach could be used to help construct the states of our MDP. Learning the structure of the problem is demonstrated by [START_REF] Strehl | Efficient structure learning in factored-state MDPs[END_REF] whose structurelearning algorithm is a variation on R-Max and [START_REF] Degris | Learning the structure of factored markov decision processes in reinforcement learning problems[END_REF] who develop SDYNA -a general framework for addressing large reinforcement learning problems by trial-and-error and no initial knowledge of their structure, and SPITI -an instantiation of SDYNA that uses incremental decision tree induction to learn the structure of a problem combined with an incre-mental version of the Structured Value Iteration algorithm. Decision tree induction allows the agent to generalize from its history (c.f. [START_REF] Bratko | Learning qualitative models[END_REF] and their qualitative controller). Features are derived by [START_REF] Guan | Modeling user feedback using a hierarchical graphical model for interactive image retrieval[END_REF] who extract objects and regions from images that the user is interested in to improve image retrieval.

Limited tests: We would have liked to have run more combinations of parameters and tested more algorithms with BIMBO. Unfortunately, a thesis has to end some time, and we therefore had to make representative choices. [START_REF] Coggan | Exploration and exploitation in reinforcement learning[END_REF] made a study of different parameters for Q-Learning and SARSA, and we were particularly interested by the idea of "disaster situation" learning. In the IE chain, this could be used if the chain goes over time with no extraction, for instance.

Due to the problem of finding a sufficiently large annotated corpus, the documents were homogeneous. They were all approximately the same length and all in the same language, English. We would have liked to have introduced more variety, but we were also lacking the services to translate and extract events in different languages. Another limitation with the corpus was that each document only contained one ideal event. We therefore did not attempt to fusion events, or to work on how to allocate feedback over multiple events. We tried using the 51 web-pages that we had extracted in section 7.4, hoping to at least test the trained AI on them, but they were so noisy (for example with html formatting, adverts, and multiple stories) that even the expert chain was unable to correctly extract the target events from them. To make use of these pages, therefore, we would need to use additional services to clean them.

Unfortunately, the lack of choice of services also impacted the possible actions that we could offer to the AI, and the observations we could make on the documents, and hence the features that we could build into our states. This meant that our state / action space was smaller than we would have liked. One of our original goals was to show the substitution of one service for another if it were better for a given type of document. We considered (but decided against) artificially degrading one of the existing services.

We currently "spoon feed" the AI the feedback at the end of the treatment. What if the AI must also learn when to ask for rewards, as well as choose its actions? Obviously, it would need to minimise the number of demands. Intuitively, it could be interesting to ask for feedback when it chooses to explore, or when it is faced with two approximately equal choices. A study would need to be made into the convergence properties.

State / action space explosion: One of the known problems with MDPs is that in a complex system, there can be a rapid explosion of the state / action space (Sutton and Barto, 1998, Chapter 8). We designed our states to have nice, neat boundaries, but in real life, often the information about the environment cannot be compartmentalised so easily. For example, if we wanted to have a feature of the state which gave the source of the document, depending on the granularity, we might need an endless list, which would be impossible to model with our current approach.

This explosion can also result in system failure, as the memory becomes insufficient. Several attempts have been made to combat this problem. [START_REF] Dean | Model minimization in Markov decision processes[END_REF] reduce the model by aggregating the states, and [START_REF] Boutilier | Stochastic dynamic programming with factored representations[END_REF] cluster states that have the same estimated value or the same optimal choice of action by representing the states using decision trees that test the values of specific variables. [START_REF] Hoey | SPUDD: Stochastic planning using decision diagrams[END_REF] developed SPUDD (Stochastic Planning Using Decision Diagrams) which uses algebraic decision diagrams to represent value functions and policies. These decision graphs are compact, and group states by their values, allowing a generalisation to be made over the structure of the problem. The use of neural networks is another avenue to explore. [START_REF] Dini | Combining q-learning with artificial neural networks in an adaptive light seeking robot[END_REF] attempt to solve the problem of scaling due to lack of space for the standard table of Q-values by using an artificial neural network as a function approximator instead (see Sutton and Barto (1998, Chapter 8) for an introduction). Their results show that for simple implementations, the standard table approach works better, but for more complex scenarios, the neural network appears to be more useful. Finally, potentially slow convergence can be tackled by using approximations of the policy iterations (for example [START_REF] Ma | A Convergent Recursive Least Squares Approximate Policy Iteration Algorithm for Multi-Dimensional Markov Decision Process with Continuous State and Action Spaces[END_REF]).

Discerning between an erroneous treatment, and faulty source information: A question that remains unanswered is how to differentiate between different types of user corrections. It may be that the treatment was flawed (which is our assumption in this manuscript), but it could also be that the original source was false, or even that the users are adding information gleaned from an external source and therefore not in the original document. Care needs to be taken that user corrections which cannot be inferred from the original document itself do not penalise the AI.

We considered allowing the system to ask for explicit feedback when the user corrects an entity, but this becomes intrusive. We can maybe draw a parallel with systems capable of filtering error-prone feedback from untrained users, for example [START_REF] Formiga | Leveraging online user feedback to improve statistical machine translation[END_REF]. They examine the use of user cor-

FUTURE WORK

157 rections to improve a statistical machine translation (SMT) system. Firstly they filter the user feedback to discard useless user translations. Secondly they align the system output and the user-edited sentence (using a variation of the Levenshtein distance on the words). Thirdly, they produce a new translation model and combine it with the one from the original system.

Future work

The limits listed above all provide interesting work for the future. We've identified several other areas that we would love to work on if we were allowed to extend this thesis indefinitely.

POMDPs and MOMDPs:

We modelled our IE chain as a "basic" MDP. The services, however, will not necessarily produce a 100% accurate result. For instance, the language is detected with a certain percentage of confidence. We could therefore see the problem as learning in a Partially Observable MDP (POMDP) where the states (e.g. "language=en_gb") are only partially observable (in fact, we only see them via the results of the extractions) and not 100% reliable or accurate (the document may contain several languages, for example). We could therefore also have the observation "language=en_us" for the same document, or even "language=fr_fr". This is particuarly pertinant, not only for the extraction of the language but for any potentially subjective or ambiguous value (e.g. the domain -politics, economics, . . . ). In this case, we might imagine using a Mixed Observability MDP (MOMDP) [START_REF] Ong | Planning under uncertainty for robotic tasks with mixed observability[END_REF] where we separate states into fully observable variables (e.g. treatment time or extraction made) and partially observable variables (e.g. language detected or domain).

Effective initialisation of the AI:

The important Q-values to be initialised from the "expert chain" concern the first (and maybe second) service(s) called -the document has to pass through a Normaliser first (to convert to an XML resource). This Normaliser could also be capable of detecting the language, but if this is not used, then the resource must pass through a Language Detector. Once we have "format" and "language" filled, we can start playing with the translation / extraction services. It's possible to create libraries of services, so we could call "a Normaliser", and pick from a list, or give them different parameters. We could therefore imagine a more general initialisation, such as : Each state where format is not detected and time / number of services under threshold gives action Normaliser; language not yet detected and time under threshold gives Language Detector, etc. This would still give a head-start to the AI, but not constrain the choice of services so much that it cannot find the optimum policy.

There's a hierarchy between the actions "choose the next service to which the resource should be sent", and the actions "choose the parameters for the service". We can treat them uniformly initially (combinatorially), but eventually, we could learn, for example, an HTN (Hierarchical Task Network), of the form : "extract named entities" decomposes into "pick the extraction service" then "choose the configuration of this extraction service" then "send resource to this extraction service". The instantiation of these sub-tasks would vary, for example which parameters to set would depend on the service, and which service to use would depend on the directories of services available. [START_REF] Kanani | Resource-bounded Information Acquisition and Learning[END_REF], for instance, treats the global actions query, download and extract. The order of these actions is fixed, but the specific contents of the action are generated dynamically. We could draw a parallel with the IE chain, where language extraction has to happen before translation, and so the translation actions are only instantiated once the language has been identified.

As we saw in section 2.3, [START_REF] Doucy | Méthodologie pour l'orchestration sémantique de services dans le domaine de la fouille de documents multimédia[END_REF] construct a modifiable document treatment chain using the WSDL of each service to ensure that the substituted service is compatible, and we could, of course use the same technique. This would also allow the AI to generalise over actions, for example, it could learn that an extraction service should follow a language detection service, as well as learning that the best language detection for a particular case is X, and the best extraction service for that case is Y. We could also use the WSDLs with the hierarchical approach to accelerate the AI's learning process by giving it more information about how the services might fit together, rather than making it find out by trial and error, thus allowing it to build prior models of the actions and states. For example, the AI would no longer try the action "extract events" when it is in a state where the language is unknown if the WSDL of the extraction module specifies that the language is required.

We could even get rid of the need to interrogate the WSDLs by using Microservice Architecture. Microservices are independently and automatically deployable, they can be built into hierarchies, and organised around the business capabilities. This allows a decentralised control, as the services no longer have to be orchestrated according to their interfaces, and is effectively what the AIs learnt to do with BIMBO's dynamic router.

We might also imagine that if we had several AIs running on different chains, that we could transfer knowledge between them. [START_REF] Singh | Transfer of learning by composing solutions of elemental sequential tasks[END_REF] invest-

FUTURE WORK

159

igates the sharing of knowledge between tasks with common sub-tasks. In our case, for instance, if the highest task was "extract events from document", we could use the knowledge gained for the sequence "collect document, clean it, detect format, detect language" which would be the same for all documents, and generalise over the states. In this way, we could have "trained" sequences or blocks of services which could be plugged into the algorithms, instead of individual services. This would also help to generalize the state space.

Reliance on the system: We saw that [START_REF] Akrour | alized Feedback: Beyond Numeric Rewards[END_REF] examine the problem of trust between the agent and the human. If the agent consistently presents trajectories that are difficult to compare, then the human is likely to become inconsistent, and the agent loses trust in them. On the other hand, if the agent is well-taught at the start and presents useful information to the human, their error-rate decreases, and the agent's trust is gained. We can imagine that if the AI is not correctly initialised, or if it is allowed to explore too much, then it might produce very poor results, such as no extractions, or ridiculous summaries. This would be counter-productive to our objective of reducing the analysts' workload, and could result in discouraging them from using the system. It would be interesting to explore the implications of providing a safety net, not only to prevent operational problems for the analyst, and possible loss of trust on both sides, but also to refine the reward process. For example, the expert chain could be run alongside the AI, and the results used to complement those of the AI. If the AI did not extract an event, but the expert chain did, then the AI could be punished, unless the analyst deletes the event extracted by the expert chain because it was false, in which case we should reward the AI.

We can, however, imagine some disadvantages to building up mutual trust.

Firstly, in seeking feedback from the analyst, we are not just capturing their knowledge, but also their preconceptions, and their subjective views. We could seek to dampen this effect by taking the views of a group of analysts, but we would obtain the preconceptions of the group as a whole instead of the individuals.

Secondly, the trust could lead to complacency. This could be the system which loses its adaptability to new situations (for example, if the exploration is reduced too much), or the analysts who rely too much on the system without using their experience, intuition and even common sense. This complacency could be a weak point, attackable by disinformation (intentional) or misinformation (unintentional). For example, the national media might be swamped with false reports of a certain candidate's popularity in order to sway voters. Another example could be the false press communiqué issued on the 22 nd November 2016 purporting to come from the group Vinci, announcing that their financial director was to be fired for fraud, and their accounts audited. Vinci suffered massive losses on the stock market that day until the fraud was discovered and denounced. Yet another example could be that of an adversary who knows that the analysts are monitoring a given situation and so creates false websites, news reports, tweets, etc. to make it seem that an attack will take place in a certain area. Once he is sure that the defensive troops are diverted there, he is free to strike elsewhere. Note that the adversary does not even need to know the model that we are using, nor how it is parametrised or trained, attacks have been successfully made which "steal" the model, enabling the adversary to duplicate its functionality [START_REF] Tramèr | Stealing machine learning models via prediction apis[END_REF].

Nor are these attacks limited to text; image recognition systems can be fooled into thinking that they see something that is not there, either by altering an image in a way that is imperceptible to the human eye [START_REF] Szegedy | Intriguing properties of neural networks[END_REF][START_REF] Kurakin | Adversarial examples in the physical world[END_REF], or through what seems like white noise or random patterns [START_REF] Nguyen | Deep neural networks are easily fooled: High confidence predictions for unrecognizable images[END_REF].

Quality improvement:

We currently treat the services as "black boxes", aiming to improve the way that the agent combines and parametrises them. We could also aim to improve the services themselves. For instance, [START_REF] Formiga | Leveraging online user feedback to improve statistical machine translation[END_REF]; [START_REF] Culotta | Corrective feedback and persistent learning for information extraction[END_REF] show that individual services can be automatically updated based on user edits. Even closer to home, [START_REF] Caron | Provenance-based quality assessment and inference in data-centric workflow executions[END_REF] and [START_REF] Amann | WebLab PROV: Computing fine-grained provenance links for XML artifacts[END_REF] improve the quality in a treatment chain by tracing the provenance of each piece of data, annotating the XML, and inferring new values. The premise is that services can propagate errors from their input data to the rest of the chain. The cumulation of errors with each service can mean a rapid degradation of quality of the end results. We can use the feedback that we gather as a quality value. As well as providing a feedback for the treatment as a whole, the feedback could also be applied to specific services in that chain, depending on the errors propagated. For instance, the quality or consistency of automatically translated text can never be higher than that of the original text, and if the detected language is false, then the consistency is bound to be low. Knowing this, we could imagine that the negative feedback is weighted to be applied to the translation service, and not to the order in which the services were chained together or parametrised. This method has the advantage of being non-intrusive, requiring no code changes to the services themselves, and can be applied to any XML based data processing workflow.

We've also mentioned that the AI's decisions can lead to interesting discoveries. [START_REF] Bratko | Learning qualitative models[END_REF] cite the importance of understanding the user's strategy, and defining it in terms that are easily accessible to the user, for example visually. It could be interesting to develop a visualisation of the thought-process behind the AI, allowing the developers, or even an interested user, to see at a glance why a certain chain configuration was chosen.

Appendix A

An example configuration

<? xml v e r s i o n=" 1 . 0 " e n c o d i n g="UTF-8" ?> <b e a n s xmlns=" h t t p : //www. s p r i n g f r a m e w o r k . o r g / schema / b e a n s "

x m l n s : x s i=" h t t p : //www. w3 . o r g / 2 0 0 1 /XMLSchema-i n s t a n c e " x m l n s : j a x w s=" h t t p : // c x f . a p a c h e . o r g / j a x w s " x m l n s : s o a p=" h t t p : // c x f . a p a c h e . o r g / b i n d i n g s / s o a p " x s i : s c h e m a L o c a t i o n=" h t t p : //www. s p r i n g f r a m e w o r k . o r g / schema / b e a n s h t t p : //www. o p e r t y name="gamma" v a l u e=" 0 . 9 " /> < !--The d e c a y p a r a m e t e r --> <p r o p e r t y name=" lambda " v a l u e=" 0 . 9 5 " /> < !--E x p l o r e i f random v a l u e l e s s than e p s i l o n --> <p r o p e r t y name=" e p s i l o n " v a l u e=" 0 . 4 " /> < !--The minimum v a l u e o f e p s i l o n , b e l o w which we do n o t d i v i d e by 2 any more --> <p r o p e r t y name=" epsilonMinimum " v a l u e=" 0 . 0 5 " /> < !--D i v i d e e p s i l o n by 2 e v e r y c h a n g e E p s i l o n b a t c h e s --> <p r o p e r t y name=" c h a n g e E p s i l o n " v a l u e=" 50 " /> < !--Do we u s e t r a c e s o r n o t ( a l w a y s u s e s newAlpha )--> <p r o p e r t y name=" useLambda " v a l u e="FALSE" /> < !--I f u s i n g t r a c e s , do we p r o f i t from t h e e x p l o r a t i o n r e s u l t s ?--> <p r o p e r t y name=" p r o f i t E x p l o r a t i o n " v a l u e="TRUE" /> < !--The l e a r n i n g r a t e ( f l a t 0 . 2 , a l p h a 0 . 5 5 ) --> <p r o p e r t y name=" l e a r n i n g R a t e " v a l u e=" 0 . 5 5 " /> < !--Are we d e c r e a s i n g e p s i l o n u s i n g t h e number o f v i s i t s t o t h e s t a t e ? --> <p r o p e r t y name=" d e c r e a s e E p s i l o n B y V i s i t s " v a l u e="TRUE" /> < !--I f so , t h e n what e x p o n e n t s h o u l d we u s e 0 . 5 < b e t a <= 1 --> <p r o p e r t y name=" e x p o n e n t " v a l u e=" 0 . 5 5 " /> </ bean> <bean i d=" C o n f i g Q L e a r n i n g " c l a s s=" c o n f i g . C o n f i g Q L e a r n i n g "> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e QLearning a l g o r i t h m </ d e s c r i p t i o n> < !--Do we u s e t h e " normal " v e r s i o n o f QLearning , but w i t h a l p h a m o d i f i e d t o t a k e i n t o a c c o u n t t h e number o f v i s i t s o r n o t --> <p r o p e r t y name=" useNewAlpha " v a l u e="TRUE" /> < !--Do we have w h o l e t r a j e c t o r i e s a s e x p l o r e / e x p l o i t ? ( n o t j u s t i n d i v i d u a l a c t i o n s ) --> <p r o p e r t y name=" w h o l e T r a j e c t o r y E x p l o r e E x p l o i t " v a l u e="FALSE" /> </ bean> <bean i d=" ConfigSSBQLearning " c l a s s=" c o n f i g . ConfigSSBQLearning "> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e SSBQLearning a l g o r i t h m </ d e s c r i p t i o n> < !--Which v e r s i o n o f p h i (WL1,WL2) t o u s e : EXP = e x p e c t a t i o n R1-R2 ; MAG = magnitude i n 1 0 0 0 , 1 0 0 , 1 , 0 , e t c . ; STR = s t r i c t i n 1 t o -1; STR1000 = s t r i c t i n 1000 t o -1000 --> <p r o p e r t y name=" p h i V e r s i o n " v a l u e="STR1000" /> < !--What p a r a m e t e r s t o u s e f o r t h e dummy w e a l t h l e v e l --> <p r o p e r t y name=" d u m m y S i m i l a r i t y " v a l u e=" 0 " /> <p r o p e r t y name="dummyTimeTaken" v a l u e=" 9999 " /> <p r o p e r t y name="magTop" v a l u e=" 30 " /> <p r o p e r t y name="magMid" v a l u e=" 20 " /> <p r o p e r t y name="magLow" v a l u e=" 10 " /> </ bean> <bean i d=" ConfigRMax " c l a s s=" c o n f i g . ConfigRMax "> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e RMax a l g o r i t h m , and t h e RMaxRandom a l g o r i t h m </ d e s c r i p t i o n> < !--The d i s c o u n t r a t e t o be u s e d f o r s o l v i n g t h e problem --> <p r o p e r t y name="gamma" v a l u e=" 0 . 9 " /> < !--The p r e c i s i o n t o which t h e problem must be s o l v e d --> <p r o p e r t y name=" e p s i l o n " v a l u e=" 1 . 0 " /> < !--The number o f t i m e s a c o u p l e ( s t a t e , a c t i o n ) w i l l be v i s i t e d b e f o r e t h e AI i s s u r e o f i t s v a l u e ( n o r m a l l y 2 0 ) --> <p r o p e r t y name=" e x p e r i e n c e " v a l u e=" 2 . 0 " /> < !--The maximum p o s s i b l e r e w a r d --> <p r o p e r t y name="rMax" v a l u e=" 1 0 0 0 . 0 " /> < !--The h o r i z o n (-1 f o r i n f i n i t y ) --> <p r o p e r t y name=" h o r i z o n " v a l u e="-1" /> </ bean> <bean i d=" ConfigVMax " c l a s s=" c o n f i g . ConfigVMax "> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e VMax a l g o r i t h m </ d e s c r i p t i o n> < !--The d i s c o u n t r a t e t o be u s e d f o r s o l v i n g t h e problem --> <p r o p e r t y name="gamma" v a l u e=" 0 . 9 5 " /> < !--The p r e c i s i o n t o which t h e problem must be s o l v e d --> <p r o p e r t y name=" e p s i l o n " v a l u e=" 0 . 0 1 " /> < !--The number o f t i m e s a c o u p l e ( s t a t e , a c t i o n ) w i l l be v i s i t e d b e f o r e t h e AI i s s u r e o f i t s v a l u e ( n o r m a l l y 2 0 ) --> <p r o p e r t y name=" e x p e r i e n c e " v a l u e=" 100 " /> < !--The maximum p o s s i b l e r e w a r d --> <p r o p e r t y name="rMax" v a l u e=" 1 0 0 0 . 0 " /> < !--The h o r i z o n (-1 f o r i n f i n i t y ) --> <p r o p e r t y name=" h o r i z o n " v a l u e="-1" /> </ bean> <bean i d=" C o n f i g S t a t e s " c l a s s=" c o n f i g . C o n f i g S t a t e s "> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e s t a t e v e c t o r </ d e s c r i p t i o n> < !--The p o s s i b l e l a n g u a g e s , i n c l u d i n g empty --> <p r o p e r t y name=" l a n g u a g e s " v a l u e=" en , " /> < !--The p o s s i b l e document f o r m a t s , i n c l u d i n g empty --> <p r o p e r t y name=" f o r m a t s " v a l u e=" t e x t / p l a i n , " /> < !--The t i m e i n t e r v a l s t h a t a r e u s e d t o c r e a t e t h e s t a t e s , f o r example , i f t h e t h r e s h o l d i s 1 8 , and t i m e I n t e r v a l i s 1 0 , t h e r e w i l l be two s t a t e s . --> < !--A l s o u s e d a s t h e margin when d e t e r m i n i n g i f two t i m e s a r e s i m i l a r o r n o t --> <p r o p e r t y name=" t i m e I n t e r v a l " v a l u e=" 5 " /> < !--The i n t e r v a l s t h a t w i l l be u s e d t o g r o u p t h e number o f s e r v i c e s p a s s e d t h r o u g h i n t o s t a t e s --> <p r o p e r t y name=" s e r v i c e I n t e r v a l s " v a l u e=" 5 , 2 0 , 100 " /> < !--Whether an i n t e r e s t i n g e v e n t h a s been e x t r a c t e d o r n o t --> <p r o p e r t y name=" i n t e r e s t i n g E v e n t E x t r a c t e d " v a l u e=" t r u e , f a l s e " /> < !--Whether any e v e n t h a s been e x t r a c t e d o r n o t --> <p r o p e r t y name=" a n y E v e n t E x t r a c t e d " v a l u e=" t r u e , f a l s e " /> </ bean> <bean i d=" C o n f i g S i m i l a r i t y W e i g h t s " c l a s s=" c o n f i g . C o n f i g S i m i l a r i t y W e i g h t s "> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e w e i g h t i n g s g i v e n t o t h e f o u r e v e n t d i m e n s i o n s and t h e c o n t e x t i n t h e c a l c u l a t i o n s o f s i m i l a r i t y </ d e s c r i p t i o n> < !--s e m a n t i c d i m e n s i o n --> <p r o p e r t y name=" semWeight " v a l u e=" 1 " /> < !--t e m p o r a l d i m e n s i o n --> <p r o p e r t y name=" temWeight " v a l u e=" 1 " /> < !--s p a t i a l d i m e n s i o n --> <p r o p e r t y name=" spaWeight " v a l u e=" 1 " /> < !--a g e n t i v e d i m e n s i o n --> <p r o p e r t y name=" agWeight " v a l u e=" 1 " /> < !--c o n t e x t u a l --> <p r o p e r t y name=" conWeight " v a l u e=" 1 " /> </ bean> <bean i d=" C o n f i g G e n e r a l " c l a s s=" c o n f i g . C o n f i g G e n e r a l "> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e program s u r r o u n d i n g t h e AI </ d e s c r i p t i o n> < !--The l e n g t h o f t i m e i n s e c o n d s t h a t w i l l be u s e d a s a p a r a m e t e r i n t h e s t a t e v e c t o r s , and t o d e t e r m i n e how l o n g t h e AI can t a k e b e f o r e b e i n g f o r c i b l y s e n t back t o t h e s t a r t and p u n i s h e d ( t h i s d e p e n d s on t h e machine , and s h o u l d be c a l i b r a t e d u s i n g t h e e x p e r t c h a i n ) --> <p r o p e r t y name=" t h r e s h o l d " v a l u e=" 10 " /> < !--The number o f documents t h a t w i l l be p r o c e s s e d b e f o r e t h e t r i p l e s t o r e i s c l e a r e d down --> <p r o p e r t y name=" c l e a r T r i p l e s t o r e " v a l u e=" 100 " /> < !--G i v e s t h e s l e e p t i m e between c h e c k s f o r t h e end o f t h e c h a i n --> <p r o p e r t y name=" s l e e p " v a l u e=" 40 " /> < !--The s e r v i c e s p l u s s

t o p t o a l l o w t h e AI t o c h o o s e t o s t o p p r o c e s s i n g t h e r e s o u r c e . T h i s i s u s e d t o d e f i n e t h e a c t i o n c h o i c e s f o r t h e AI ( 0 = t i k a ,

1 = ngramj , 2 = geo , 3 = GATE) , e t c . --> <p r o p e r t y name=" s e r v i c e s A n d S t o p " v a l u e=" d i r e c t : t i k a , d i r e c t : n g r a m j , d i r e c t : g a t e , d i r e c t : g e o , s t o p " /> < !--Whether t h e WebLab r e s o u r c e s h o u l d be i n d e x e d o r n o t --> <p r o p e r t y name=" i n d e x R e s o u r c e " v a l u e="FALSE" /> < !--Whether t h e t r i p l e s t o r e s h o u l d be u s e d o r n o t . I f FALSE , t h e n t h e i n t e r n a l Jena I n f M o d e l i s u s e d ( f o r b e t t e r p e r f o r m a n c e ) --> <p r o p e r t y name=" u s e T r i p l e s t o r e " v a l u e="FALSE" /> < !--Whether t h e s i m i l a r i t y r e s u l t s s h o u l d be s t o r e d o r n o t --> < !--The b a s e name o f t h e c o n s o l e d a t f i l e w r i t t e n a t e a c h i t e r a t i o n , and a t t h e end --> <p r o p e r t y name=" c o n s o l e F i l e n a m e " v a l u e=" console_summary " /> < !--The b a s e name o f t h e r e w a r d d a t f i l e w r i t t e n a t e a c h i t e r a t i o n , and a t t h e end --> <p r o p e r t y name=" r e w a r d F i l e n a m e " v a l u e=" r e w a r d s " /> < !--The b a s e name o f t h e e x p l o r e / e x p l o i t d a t f i l e w r i t t e n a t e a c h i t e r a t i o n , and a t t h e end --> <p r o p e r t y name=" e x p l o r e E x p l o i t F i l e n a m e " v a l u e=" e x p l o r e E x p l o i t " /> < !--The f i l e c o n t a i n i n g t h e s e r i a l i z e d GTD e v e n t s --> <p r o p e r t y name=" g t d E v e n t s S e r F i l e n a m e " v a l u e=" g t d E v e n t s M o d i f i e d . s e r " /> < !--The f i l e c o n t a i n i n g t h e e v e n t s e x t r a c t e d by BIMBO --> <p r o p e r t y name=" e v e n t s E x t r a c t e d F i l e n a m e " v a l u e=" e v e n t s E x t r a c t e d " /> < !--The f i l e c o n t a i n i n g ( s , a , s prime , r ) , u s e d t o c o n s t r u c t d o t f i l e f o r g r a p h v i z --> <p r o p e r t y name=" t r a n s i t i o n R e w a r d s F i l e n a m e " v a l u e=" t r a n s i t i o n R e w a r d s " /> </ bean> <bean i d=" C o n f i g G a z e t t e e r s " c l a s s=" c o n f i g . C o n f i g G a z e t t e e r s "> < d e s c r i p t i o n> C o n t a i n s t h e c o n f i g u r a t i o n p a r a m e t e r s f o r t h e GATE g a z e t t e e r s t h a t t h e AI c o n t r o l s </ d e s c r i p t i o n> < !--F i l e names o f t h e nouns and v e r b s g a z e t t e e r s --> <p r o p e r t y name=" g a z e t t e e r N o u n s F i l e n a m e s " v a l u e=" l i s t E v e n t N o u n s B o m b i n g O n l y . d e f , l i s t E v e n t N o u n s I n j u r e O n l y . d e f , listEventNounsDummyOnly . d e f , l i s t E v e n t N o u n s S h o o t i n g O n l y . d e f , l i s t E v e n t N o u n s M i x e d . d e f " /> <p r o p e r t y name=" g a z e t t e e r V e r b s F i l e n a m e s " v a l u e=" l i s t E v e n t V e r b s B o m b i n g O n l y . d e f , l i s t E v e n t V e r b s I n j u r e O n l y . d e f , listEventVerbsDummyOnly . d e f , l i s t E v e n t V e r b s S h o o t i n g O n l y . d e f , l i s t E v e n t V e r b s M i x e d . d e f " /> < !--F i l e names o f t h e nouns and v e r b s m a s t e r l i s t s --> <p r o p e r t y name=" g a z e t t e e r N o u n s M a s t e r F i l e s " v a l u e=" l i s t E v e n t N o u n s . d e f , l i s t E v e n t N o u n s E m p t y . d e f " /> <p r o p e r t y name=" g a z e t t e e r V e r b s M a s t e r F i l e s " v a l u e=" l i s t E v e n t V e r b s . d e f , l i s t E v e n t V e r b s E m p t y . d e f " /> < !--L o c a t i o n o f t h e nouns and v e r b s l i s t s --> <p r o p e r t y name=" n o u n s L i s t L o c a t i o n " v a l u e=" EventNouns / l i s t E v e n t N o u n s . d e f " /> <p r o p e r t y name=" v e r b s L i s t L o c a t i o n " v a l u e=" EventVerbs / l i s t E v e n t V e r b s . d e f " /> < !--Which g a z e t t e e r s s h o u l d be u s e d when r e s e t t i n g GATE --> <p r o p e r t y name=" r e s e t G a z e t t e e r N o u n s " v a l u e=" l i s t E v e n t N o u n s E m p t y . d e f " /> <p r o p e r t y name=" r e s e t G a z e t t e e r V e r b s " v a l u e=" l i s t E v e n t V e r b s E m p t y . d e f " /> < !--Which e v e n t i s t h e i n t e r e s t i n g one , g e t t i n g e x t r a r e w a r d s --> <p r o p e r t y name=" i n t e r e s t i n g E v e n t " v a l u e=" h t t p : // weblab . ow2 . o r g / w o o k i e# BombingEvent " /> </ bean> </ b e a n s> Listing A.1 -An example configuration file.

Chain and dynamic router

<? xml v e r s i o n=" 1 . 0 " e n c o d i n g="UTF-8" ?> <b e a n s xmlns=" h t t p : //www. s p r i n g f r a m e w o r k . o r g / schema / b e a n s "

x m l n s : x s i=" h t t p : //www. w3 . o r g / 2 0 0 1 /XMLSchema-i n s t a n c e " x m l n s : j a x w s=" h t t p : // c x f . a p a c h e . o r g / j a x w s " x m l n s : s o a p=" h t t p : // c x f . a p a c h e . o r g / b i n d i n g s / s o a p " x m l n s : c a m e l=" h t t p : // c a m e l . a p a c h e . o r g / schema / s p r i n g " x m l n s : c x f=" h t t p : // c a m e l . a p a c h e . o r g / schema / c x f " x m l n s : o s g i=" h t t p : //www. s p r i n g f r a m e w o r k . o r g / schema / o s g i " x s i : s c h e m a L o c a t i o n=" h t t p : //www. s p r i n g f r a m e w o r k . o r g / schema / b e a n s h t t p : //www. s p r i n g f r a m e w o r k . o r g / schema / b e a n s / s p r i n g -beans -3 . 0 . xsd h t t p : // c a m e l . a p a c h e . o r g / schema / s p r i n g h t t p : // c a m e l . a p a c h e . o r g / schema / s p r i n g / camel-s p r i n g . xsd h t t p : //www. s p r i n g f r a m e w o r k . o r g / schema / o s g i h t t p : //www. s p r i n g f r a m e w o r k . o r g / schema / o s g i / s p r i n g -o s g i . xsd h t t p : // c x f . a p a c h e . o r g / j a x w s h t t p : // c x f . a p a c h e . o r g / schemas / j a x w s . xsd h t t p : // c a m e l . a p a c h e . o r g / schema / c x f h t t p : // c a m e l . a p a c h e . o r g / schema / c x f / camelc x f . xsd "> <bean i d=" bimbo " c l a s s=" bimbo . Bimbo" /> <bean i d=" f i l e r e a d e r " c l a s s=" p r o c e s s o r . F i l e R e a d e r P r o c e s s o r " /> < !--======================================================= --> < !--===================== Web s e r v i c e s ==================== --> < !--======================================================= --> < o s g i : s e r v i c e i d=" o s g i -s e r v i c e -t i k a " r e f=" t i k a " i n t e r f a c e=" o r g . a p a c h e . c a m e l . En d poi n t "> < o s g i : s e r v i c e -p r o p e r t i e s> <e n t r y key="name" v a l u e=" s e r v i c e -t i k a " /> </ o s g i : s e r v i c e -p r o p e r t i e s> </ o s g i : s e r v i c e> < o s g i : s e r v i c e i d=" o s g i -s e r v i c e -ngramj " r e f=" ngramj " i n t e r f a c e=" o r g . a p a c h e . c a m e l . En d poi n t "> < o s g i : s e r v i c e -p r o p e r t i e s> <e n t r y key="name" v a l u e=" s e r v i c e -ngramj " /> </ o s g i : s e r v i c e -p r o p e r t i e s> </ o s g i : s e r v i c e> < o s g i : s e r v i c e i d=" o s g i -s e r v i c e -g a t e " r e f=" wookie-g a t e " i n t e r f a c e=" o r g . a p a c h e . c a m e l . En d poi n t "> < o s g i : s e r v i c e -p r o p e r t i e s> <e n t r y key="name" v a l u e=" s e r v i c e -g a t e " /> </ o s g i : s e r v i c e -p r o p e r t i e s> </ o s g i : s e r v i c e> < o s g i : s e r v i c e i d=" o s g i -s e r v i c e -g e o " r e f=" g e o "

i n t e r f a c e=" o r g . a p a c h e . c a m e l . En d poi n t "> < o s g i : s e r v i c e -p r o p e r t i e s> <e n t r y key="name" v a l u e=" s e r v i c e -g e o " /> </ o s g i : s e r v i c e -p r o p e r t i e s> </ o s g i : s e r v i c e> o n s t a n t> </ s e t H e a d e r> </ f i l t e r > <t o u r i=" w e b l a b : // c r e a t e ? t y p e=Document&amp ; outputMethod=xml " /> <convertBodyTo t y p e=" j a v a . l a n g . S t r i n g " /> <t o u r i=" d i r e c t : s t a r t " /> </ r o u t e> The attacks come even as the Union government is all set to launch a massive offensive against Left wing extremism in the naxalite-hit States

The violence occurred on the first day of a two-day shutdown called by the Communist Party of India (Maoist).

The body of Jharkhand Party member Kanai Murmu, who was kidnapped from his Ergada village in the early hours of Monday, was found later, the police said.

Informed sources said another member of the party, Ananda Mahato, was abducted by Maoists from Kulabheda village in the same Midnapore district. Both Murmu and Mahato were members of the anti-Maoist "Maobadi Protirodh Committee." At Ghatbera in Purulia district of West Bengal, Maoists ransacked the house of a local CPI(Marxist) leader and shot and injured a member of the village resistance committee. As the attackers failed to find Chandan Singh Laya, CPI(Marxist) Ghatbera local committee secretary, they set his house ablaze.

In Jharkhand, Maoists blew up rail tracks at Jharandih in Dhanbad, resulting in the Shaktipunj Express and some local trains being held up at various points, Senior Public Relations Officer of Dhanbad Rail Division Amrendra Das said adding a light engine derailed after a one-and-half metre section of the track was blown up.

A group of 12 Maoists set three trucks ablaze in Giridih district's Isri area and blocked the Dumri-Giridih road with felled trees. Maoists also partially blasted a road bridge connecting Dumri to the Grand Trunk Road and gunshots were heard, Giridih Superintendent of Police Ravi Kant Dhan said.

Hazaribagh Superintendent of Police Pankaj Kamboj said Maoists partially damaged a road with explosives at Sardalo, bordering Bokaro district.

In Bihar, Maoists blasted the telecom tower at Salaiya village. Maoists also dug up a 15-metre stretch of a road at Chanda village, disrupting traffic between Deo and Dhibra. They left behind pamphlets, claiming responsibility for the action, official sources said. . o r g / s e r v i c e s#L a n g u a g e E x t r a c t i o n " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> </ rdf:RDF> </ d a t a> </ a n n o t a t i o n> <a n n o t a t i o n u r i=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#a66 "> <d a t a x m l n s : w l=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / model#" x m l n s : r e s o u r c e C o n t a i n e r=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / s e r v i c e s / r e s o u r c e c o n t a i n e r "> <rdf:RDF x m l n s : r d f=" h t t p : //www. w3 . o r g /1999/02/22 -r d f -s y n t a x-n s#"> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // sws . geonames . o r g / 1 2 2 7 6 0 3 / "> < r d f : t y p e r d f : r e s o u r c e=" h t t p : //www. geonames . o r g / o n t o l o g y# F e a t u r e " /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 9 5 6 3 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 5 5 1 4 7 / " x m l n s : g e o=" h t t p : //www. geonames . <g e o : h a s S p a t i a l T h i n g r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 6 7 2 5 4 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#k a r n a t a k a "> <g e o : h a s S p a t i a l T h i n g r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 6 7 7 0 1 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // sws . geonames . o r g / 1 2 7 3 2 9 3 / "> < r d f : t y p e r d f : r e s o u r c e=" h t t p : //www. geonames . o r g / o n t o l o g y# F e a t u r e " /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 9 5 6 3 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 5 5 1 4 7 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 6 9 7 5 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d f -schema# ">N a t i o n a l C a p i t a l T e r r i t o r y o f D e l h i</ r d f s : l a b e l> <g e o : n a m e x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#"> N a t i o n a l C a p i t a l T e r r i t o r y o f D e l h i</ g e o : n a m e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // sws . geonames . o r g / 6 2 9 5 6 3 0 / "> < r d f : t y p e r d f : r e s o u r c e=" h t t p : //www. geonames . o r g / o n t o l o g y# F e a t u r e " /> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d f -schema# ">Ear th</ r d f s : l a b e l> <g e o : n a m e x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#">Earth </ g e o : n a m e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#w e s t _ b e n g a l "> <g e o : h a s S p a t i a l T h i n g r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 5 2 8 8 1 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#d e l h i "> <g e o : h a s S p a t i a l T h i n g r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 7 3 2 9 3 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#i n d i a "> <g e o : h a s S p a t i a l T h i n g r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 6 9 7 5 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // sws . geonames . o r g / 1 2 6 7 2 5 4 / "> < r d f : t y p e r d f : r e s o u r c e=" h t t p : //www. geonames . o r g / o n t o l o g y# F e a t u r e " /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 9 5 6 3 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 5 5 1 4 7 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 6 9 7 5 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d f -schema# ">S t a t e o f K e r a l a</ r d f s : l a b e l> <g e o : n a m e x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#">S t a t e o f K e r a l a</ g e o : n a m e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // sws . geonames . o r g / 1 2 6 7 7 0 1 / "> < r d f : t y p e r d f : r e s o u r c e=" h t t p : //www. geonames . o r g / o n t o l o g y# F e a t u r e " /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 9 5 6 3 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 5 5 1 4 7 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 6 9 7 5 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d f -schema# ">S t a t e o f KarnÄĄtaka</ r d f s : <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 9 5 6 3 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 5 5 1 4 7 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 6 9 7 5 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d f -schema# ">S t a t e o f West B e n g a l</ r d f s : l a b e l> <g e o : n a m e x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#">S t a t e o f West B e n g a l</ g e o : n a m e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // sws . geonames . o r g / 1 2 6 9 7 5 0 / "> < r d f : t y p e r d f : r e s o u r c e=" h t t p : //www. geonames . o r g / o n t o l o g y# F e a t u r e " /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 9 5 6 3 0 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> <g e o : p a r e n t F e a t u r e r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 6 2 5 5 1 4 7 / " x m l n s : g e o=" h t t p : //www. geonames . </ r d f : D e s c r i p t i o n > <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="18" end ="52" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-28"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="248" end ="257" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-38"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="261" end ="267" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-39"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="271" end ="280" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-36"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="502" end ="513" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-16"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="526" end ="531" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-37"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="626" end ="632" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-60"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="654" end ="665" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-17"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="813" end ="835" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-27"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="840" end ="846" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-59"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="850" end ="861" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-14"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="965" end ="970" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-15"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1004" end ="1020" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-26"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1196" end ="1220" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-21"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1246" end ="1280" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-20"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1290" end ="1299" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-58"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1309" end ="1323" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-4"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1486" end ="1503" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-5"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1516" end ="1534" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-2"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1591" end ="1618" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-19"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1636" end ="1652" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-3"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1656" end ="1667" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-8"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="1838" end ="1848" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-46"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2122" end ="2156" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-18"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2307" end ="2323" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-9"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2527" end ="2533" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-45"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2534" end ="2548" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-47"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2586" end ="2592" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-50"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2593" end ="2606" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-44"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2630" end ="2637" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-56"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2683" end ="2698" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-6"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2705" end ="2710" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-42"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2749" end ="2764" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-7"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2818" end ="2832" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-53"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="2971" end ="2976" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-43"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="3008" end ="3014" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-51"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="3310" end ="3319" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-40"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="3323" end ="3329" u r i =" h t B r e a k i n g News | News Update S t o r i e s i n t h i s S e c t i o n M a o i s t s u n l e a s h v i o l e n c e i n West Bengal , Jharkhand , B i h a r J o i n t o p e r a t i o n s t o c o n t i n u e : Buddhadeb I n d u s t r i a l growth s o a r e d t o a h i g h i n August Ostrom , W i l l i a m s o n win Economics Nobel They showed t h a t e c o n o m i c a n a l y s i s can s h e d l i g h t on s o c i a l o r g a n i s a t i o n L a s h k a r c h i e f ' s r e l e a s e c a u s e o f g l o b a l c o n c e r n L a h o r e c o u r t q u a s h e s two c a s e s a g a i n s t S a e e d MPs team v i s i t s h i l l r e g i o n i n S r i Lanka , i n t e r a c t s w i t h I n d i a n -o r i g i n w o r k e r s I n d i a t o t a k e up S a e e d ' s c a s e w i t h P a k i s t a n P r i t h v i -I I m i s s i l e s s u c c e s s f u l l y t e s t -f i r e d V i l l a g e r s want D i n a k a r a n ' s s u r p l u s l a n d h o l d i n g s r e d i s t r i b u t e d t o them "No c h a n n e l opened w i t h V i j a y a k a n t Ajmal wants t r i a l h e l d i n i n t e r n a t i o n a l c o u r t Rs . 2 ,000 -c r .

Bihar Additional

Annexe F

Résumé en français

F.1 Introduction

Imaginons une usine qui produit des voitures avec une roue sur le toit, une sur le côté, et deux qui touchent le sol. Les clients de cette usine étrange doivent changer les roues à chaque fois qu'ils achètent une voiture. Cela peut sembler improbable, mais c'est le cas avec certains systèmes d'Extraction d'Information (EI). D'énormes quantités de données sont traitées, et le système tente de fournir une synthèse des informations utiles. Des failles dans l'extraction de cette information rendent la synthèse inexacte, et c'est à l'utilisateur de la corriger.

L'usine se rend compte que les clients ne sont pas contents, donc elle met à leur disposition un expert qui peut produire des voitures sur demande. Le client n'a qu'à trouver le temps de venir dans l'usine pour discuter de ses besoins précis. D'une façon similaire, le système EI peut être configuré par un expert en consultation avec l'utilisateur (en espérant que l'utilisateur ne change pas d'avis une fois qu'il voit les résultats).

L'usine décide d'améliorer la satisfaction des utilisateurs, qui ont maintenant des outils leur permettant de changer la configuration de l'usine euxmêmes après une formation en assemblage de voitures. Les utilisateurs des systèmes EI ont des langages développés spécifiquement qui leur permettent d'interagir directement avec la chaîne, mais il peut leur manquer l'expertise pour apprécier les conséquences de leurs actions.

L'usine prend une décision radicale. Elle crée un département dédié aux retours utilisateur. Elle fait des sondages, demandant aux utilisateurs de noter les caractéristiques de leurs voitures de 1 à 1000, ou de les trier en ordre de préférence. Elle utilise ensuite ces questionnaires pour construire des voitures fabriquées individuellement pour chaque client. Nous pouvons voir que l'obtention des retours utilisateur pourrait aider à améliorer les services EI, mais est-ce vraiment naturel de demander qu'ils fournissent ces retours explicitement et de façon numérique ?

Enfin, l'usine installe un dispositif de contrôle continu dans la voiture. Cet outil fait automatiquement des retours au département sur l'utilisation 200 ANNEXE F. RÉSUMÉ EN FRANÇAIS exacte de la voiture, les modifications effectuées, la couleur dans laquelle le client choisit de repeindre sa voiture, quelles caractéristiques lui plaisent le plus, etc. Enfin, elle peut produire des voitures uniques qui plaisent aux clients, sans avoir besoin de les interroger, les former, ou demander qu'ils remplissent de longs formulaires.

Notre objectif est d'installer un tel dispositif de contrôle continu dans la chaîne EI. Il observera les actions de l'utilisateur, et n'aura besoin que de retours intuitifs et naturels pour améliorer la chaîne (mais bien sûr, il pourra aussi prendre en compte des retours numériques).

F.1.1 Contexte

Notre travail est motivé par les besoins réels de la communauté Renseignement d'Origine Source Ouverte (ROSO). Par le passé, la tâche des analystes des services de renseignement a été de chercher des informations cachées. Maintenant, ils font face à un volume toujours croissant de pages web, ainsi que d'autres documents d'origine source ouverte, et ils doivent extraire de l'information utile et pertinente à partir de cette profusion de données. Des chaînes de traitement spécialisées ont été développées pour faciliter cette tâche (par exemple, Ogrodniczuk and Przepiórkowski (2010) donnent une synthèse de quelques chaînes fournies en tant que services Web).

Nous nous intéressons au problème générique de l'amélioration continue d'une telle chaîne de traitement de documents, plus précisément d'extraction d'événements d'intérêt pour la veille et le renseignement. Dans cette application, des documents provenant essentiellement du Web sont fournis en continu à une chaîne de traitement, qui vise à extraire des événements (par exemple, une attaque terroriste) et leurs caractéristiques (date, lieu, acteurs, etc.) pour les intégrer dans une base de données.

Dans de telles applications, il est clair que l'extraction ne peut pas être parfaite. Tout d'abord, du fait de la grande diversité de documents sur le Web, il ne peut pas exister une unique chaîne optimale pour tous les documents. Ensuite, on peut imaginer, par exemple, qu'une dépêche relatant « le bombardement, par des ions, d'une cible d'or dans une animation autour de la physique atomique lors d'une manifestation pour la fête de la science », puisse induire une chaîne de traitement en erreur, et lui fasse insérer dans la base un attentat à l'arme atomique. Par ailleurs, la volonté de traiter des documents provenant du monde entier entraîne le besoin de traiter des documents dans des langues très diverses, pour lesquelles des dictionnaires peuvent être de qualité très variable. Pour toutes ces raisons, les analystes perdent du temps dans la tâche lourde et répétitive de la correction a poste- L'application qui nous intéresse utilise une chaîne de traitement, définie par des experts, consistant en un enchaînement figé (mais potentiellement conditionnel) de traitements atomiques, tels que la détection de la langue ou du format, la traduction, la détection d'événements en utilisant des mots ou verbes déclencheurs, etc. Actuellement, aucun mécanisme ne permet d'améliorer la chaîne au fil du temps sans l'intervention d'un expert, en consultation avec les analystes, pour recalibrer la chaîne.

Notre objectif est de combler ce manque, en fournissant un mécanisme automatique d'amélioration continue de la chaîne de traitement, tirant parti des retours (ou feedback ) exprimés implicitement par les analystes lorsqu'ils consultent la sortie du système. Il s'agit de faire en sorte que la chaîne « apprenne de ses erreurs ». Par exemple, si le document est un article scientifique en français, il peut être préférable de le traduire d'abord en anglais, pour profiter du plus grand choix de règles d'extraction pour l'anglais, puis d'extraire les événements. En revanche, pour un blog sur lequel l'argot est abondamment utilisé, une extraction directe peut être préférable, car les dictionnaires sont insuffisants pour l'argot, et une traduction ne fait qu'ajouter du bruit. Ainsi, les services, ou briques pour la construction de la chaîne, restent des « boîtes noires » permettant à l'analyste de se distancier du processus d'extraction.

Nous devons récupérer le feedback de manière invisible pour l'analyste, sans impact sur son travail. [START_REF] Bratko | Learning qualitative models[END_REF] montrent que l'expertise de l'utilisateur (son feedback ) est manifestée par les traces de ses actions. Ces traces pourraient être captées à travers l'interface graphique qui donne le détail des événements en synthèse, et le feedback basé sur les interactions analyste-système actuelles. Par exemple, d'un événement corrigé, nous pouvons tirer un feedback explicite, via une distance entre l'événement idéal (corrigé) et celui qui avait été extrait. Nos premières expériences simulent ceci : l'agent reçoit un jugement sur la qualité de ses extractions sous la forme d'une distance, et donc d'un feedback numérique.

Mais l'événement peut aussi être non extrait, correct mais extrait par un traitement trop lent, etc. Prendre en compte ces différents aspects donne lieu à des questions, telles que « Est-il préférable d'extraire un événement incorrect, ou de manquer une extraction ? La qualité est-elle plus importante que la vitesse, ou l'inverse ? ». Il n'est pas évident de répondre à de telles questions avec une réponse numérique, par exemple, de dire qu'une extraction partielle mais rapide vaut 10.5 fois moins qu'une extraction complète mais lente. Il est essentiel, alors, que nous progressions vers un feedback intuitif, basé sur des préférences que les utilisateurs puissent exprimer naturellement, 202 ANNEXE F. RÉSUMÉ EN FRANÇAIS telles que « Je préfère une extraction incorrecte à une extraction manquée », ou encore « Je préfère que cela soit aussi rapide que possible ». Les besoins peuvent alors être spécifiés et changés très facilement, de façon intuitive.

F.1.2 Contribution

Nous formalisons notre problème comme un problème d'apprentissage par renforcement, et la chaîne de traitement elle-même comme un processus de décision markovien (MDP, Définition 2). Nous étudions trois cadres différents. Dans le premier, l'agent reçoit des retours numériques sur la qualité du traitement, ce qui est très informatif, mais difficile à calibrer. Les deux autres cadres utilisent des retours intuitifs (pour l'utilisateur), avec une formalisation partiellement qualitative pour le deuxième, et totalement qualitative pour le troisième. De tels retours sont moins informatifs pour l'agent, mais requièrent beaucoup moins, voire pas du tout de calibrage.

Nous rendons compte de la plateforme que nous avons construite, appelée BIMBO (Benefiting from Intelligent and Measurable Behaviour Optimisation). Dans cette plateforme, on peut « brancher » différents algorithmes d'apprentissage par renforcement, différents modèles de la tâche, différents services web, différents types de retours de l'utilisateur, etc., ce qui nous permet de mesurer l'impact de ces éléments sur l'apprentissage. Nous rendons ensuite compte de l'utilisation de cette plateforme pour l'application industrielle qui nous intéresse, c'est-à-dire pour l'amélioration continue d'une chaîne de traitement qui extrait des événements à partir de pages web et d'autres documents de sources ouvertes. La plateforme sous-tend notre application, mais peut également être utilisée pour d'autres applications, et/ou pour l'étude de différentes méthodes d'amélioration continue de chaînes de traitement. Nous décrivons brièvement, en conclusion, son utilisation pour des tâches de segmentation d'images et de reconnaissance optique de caractères.

Nous présentons ensuite des résultats expérimentaux, tout d'abord avec des retours numériques (simulés) des utilisateurs (section F.7). Les résultats montrent qu'apprendre automatiquement à enchaîner et paramétrer les services est totalement faisable en pratique. Nous présentons enfin des résultats expérimentaux avec des retours intuitifs (section F.8). Là encore, et malgré l'information bien moins précise véhiculée par de tels retours, les résultats montrent que l'apprentissage reste totalement faisable en pratique, et ce, en demandant très peu d'effort de calibrage de la part des analystes.

Notre approche, aussi, peut être utilisée pour d'autres traitements, par exemple nous l'avons appliqué, en utilisant la plateforme BIMBO et une va-F.1. INTRODUCTION 203 riation de Q-Learning(λ), Dora [START_REF] Nicart | Dora Qlearning -making better use of explorations[END_REF], aux tâches de segmentation (le pré-traitement de l'image) et de la ROC (reconnaissance optique de caractères). Notre but est la détection des éléments d'intérêt, pour suggérer automatiquement des titres depuis les pages de garde des documents.

F.1.3 Travaux connexes

À notre connaissance, il n'y a pas eu de travaux académiques s'intéressant à des agents capables de construire dynamiquement une chaîne de services, et d'apprendre et de s'améliorer en continu en utilisant des retours intuitifs des utilisateurs. Toutefois, de nombreux ingrédients ont été étudiés et proposés pour de telles applications.

L'enchaînement de services pour réaliser un but précis, par exemple, ne constitue pas une idée nouvelle. Des services peuvent être composés, par exemple, pour aider l'utilisateur à remplir des formulaires [START_REF] Saïs | Proceedings of SOS-DLWD 2013[END_REF]. Dans une autre application, les caractéristiques physiques de documents sont utilisées avec succès pour construire, en temps réel, des chaînes adaptatives et modulaires pour des photocopieurs Xerox [START_REF] Fromherz | Model-based computing for design and control of reconfigurable systems[END_REF]. Toutefois, les entrées et sorties de chaque service sont connues à l'avance, et enchaîner ces services constitue donc un problème de planification, sans apprentissage. Par ailleurs, les préférences de l'utilisateur ne sont pas prises en compte, et aucun retour n'est utilisé pour améliorer le processus d'une fois sur l'autre.

De façon générale, un utilisateur peut reconfigurer une chaîne en utilisant un langage dédié, comme ReCooPLa [START_REF] Rodrigues | Towards an engine for coordination-based architectural reconfigurations[END_REF], ou en choisissant un service dans un répertoire de services du même type [START_REF] Doucy | Méthodologie pour l'orchestration sémantique de services dans le domaine de la fouille de documents multimédia[END_REF]. Néanmoins, une chaîne ainsi reconfigurée ne s'adapte pas dynamiquement sans intervention de l'utilisateur, et ce dernier doit avoir une certaine expertise sur le système pour apprécier les conséquences de ses choix.

L'utilisation de retours de l'utilisateur, implicites ou explicites, pour l'apprentissage par renforcement, a également été explorée. C'est notamment l'objet d'un thème très actuel en intelligence artificielle : l'apprentissage par renforcement inverse. Par exemple, [START_REF] Knox | Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and performance[END_REF] étudient, sur des tâches synthétiques mais avec des utilisateurs réels, l'influence de la polarité (« encourageant », « punisseur », etc.) d'un utilisateur entraînant un agent, via des retours continuels, pour des tâches séquentielles. [START_REF] Loftin | Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning[END_REF] étudient l'inférence de feedback implicite, non numérique, par des agents à partir de l'observation des actions prises par des utilisateurs sur les mêmes tâches. Un autre exemple de retour implicite consiste, pour l'utilisateur, à indiquer sa préférence parmi deux politiques montrées par l'agent [START_REF] Akrour | Preference-based policy learning[END_REF]. Par ailleurs, des modèles du comportement [START_REF] Azaria | Strategic advice provision in repeated human-agent interactions[END_REF] et des préférences [START_REF] Karami | Apprentissage de connaissances d'adaptation à partir des feedbacks des utilisateurs[END_REF] d'un utilisateur peuvent être construits en utilisant des interactions utilisateur-système, pour fournir des systèmes personnalisés. Toutefois, bien que ces systèmes soient capables de bien réagir, en particulier à des changements d'utilisateurs et de préférences, ils ne sont pas adaptés à notre cas d'utilisation, car ils requièrent une interaction continue avec l'utilisateur, qui doit constamment critiquer les choix de l'agent. Ceci aurait un coût prohibitif dans notre application, car les utilisateurs principaux du système sont les analystes, dont l'amélioration du système d'extraction n'est pas la tâche première, et car la chaîne est appelée à traiter des volumes extrêmement importants de documents (de l'ordre d'un document toutes les 20 secondes, en continu, pour une instance de la chaîne).

Nous considérons actuellement les services individuels constituant la chaîne comme des « boîtes noires », avec pour objectif que l'agent améliore la façon de les enchaîner et de les paramétrer. Nous notons toutefois que des travaux s'intéressent à l'estimation de la qualité de tels services [START_REF] Caron | Provenance-based quality assessment and inference in data-centric workflow executions[END_REF] et aux dégradations potentielles dans la chaîne [START_REF] Amann | WebLab PROV: Computing fine-grained provenance links for XML artifacts[END_REF], ainsi qu'à leur mise à jour automatique en utilisant les retours de l'utilisateur [START_REF] Formiga | Leveraging online user feedback to improve statistical machine translation[END_REF]. On pourrait tout à fait utiliser une combinaison de ces techniques en parallèle de notre approche.

F.2 La plateforme WebLab

La chaîne de traitement qui motive notre travail s'appuie sur la plateforme open-source (WebLab, 2016c). WebLab intègre des services web qui encapsulent les processus unitaires, et qui peuvent être interchangés ou permutés afin de créer une chaîne de traitement. Cette chaîne peut ensuite être utilisée pour analyser des documents multimédia open-source, et en extraire l'information.

Une chaîne de traitement typique de WebLab (Figure F.1), quand elle traite un nouveau document, commence par convertir le document source en une ressource XML. Cette ressource est ensuite transmise de service en service. Chaque service analyse le contenu de la ressource telle qu'il la reçoit, et l'enrichit avec des annotations. Enfin, les résultats sont stockés pour consultation par l'analyste. • T ⊆ T est la dimension temporelle de E, c'est-à-dire quand E est survenu (potentiellement ambiguë, telle que « mardi dernier ») ;

• G ⊆ G est la dimension spatiale de E, c'est-à-dire le lieu où E est survenu (potentiellement ambiguë également) ;

• A ⊆ A est la dimension agentive de E, c'est-à-dire l'ensemble des participants impliqués.

Plus précisément : • C = {AttackEvent, BombingEvent} ;

• T = {4/29/1971} ;

• G = {Los Angeles, California, United States, US, North America} ;

• A = {Chicano Liberation Front, Bank of America}.

L'efficience d'une chaîne de traitement ne dépend pas seulement des services utilisés et de l'ordre dans lequel ils sont appelés, mais elle dépend aussi des paramètres utilisés par ces services. Par exemple, l'extracteur GATE est paramétré par des dictionnaires (gazetteers), qui sont des listes de noms et de verbes déclenchant la détection d'un type spécifique d'événement. Son dictionnaire de verbes pour l'extraction d'événements de type bombing contient typiquement les verbes explode, detonate, etc. Nous voyons donc GATE comme un service paramétré : choisir les bons dictionnaires à lui donner en paramètre fait partie des actions à (apprendre à) réaliser pour l'agent.

F.3 Apprentissage par renforcement

Dans la pratique, la chaîne de traitement utilisée est complexe. Elle est écrite et calibrée par des experts qui choisissent les services constituant la chaîne, leur ordonnancement et leurs paramètres (par exemple, les gazetteers de mots déclencheurs pour les services de détection d'événements et d'entités nommées). L'ordonnancement est figé, tout en pouvant être conditionnel (par exemple, si le document est en format PDF, passer au service 1 pour le convertir en XML, et passer au service 2 sinon).

Même avec de l'expertise, il est très difficile d'obtenir une chaîne parfaite. Ceci provient du fait que l'utilisation des documents open source du Web apporte des difficultés : leurs formats et contenus ne sont pas standardisés, les pages sources elles-mêmes ne sont pas contrôlables, elles peuvent comporter du « bruit » (comme de la publicité), être piratées, et leurs URL peuvent changer. On observe des erreurs d'extraction pouvant consister en des événements d'intérêt entièrement ou partiellement manqués, ou des événements 208 ANNEXE F. RÉSUMÉ EN FRANÇAIS mal extraits (des informations non connexes dans la même phrase associées faussement, par exemple).

Par exemple, imaginons que l'analyste veuille de l'information sur les accords entre pays. Il est impossible de dire avec certitude si le mot « alliance » dans un document y fait référence ou non. Ce n'est qu'après l'extraction de l'événement, déclenchée par le mot « alliance », que l'analyste peut se rendre compte que la page parle de mariages, par exemple. Même si le document provient d'un journal politique, il se peut que l'on parle d'une coalition entre partis politiques, ou que les filtres de publicité n'aient pas réussi à attraper une vente de bagues. Un synonyme tel qu'« union » a pu être utilisé au lieu d'« alliance », et l'événement n'a pas été reconnu. Avec ces incertitudes, les experts qui paramètrent la chaîne essaient d'envisager les situations les plus communes. Il est inconcevable qu'ils puissent construire des chaînes à la main en examinant chaque document source.

Les seules informations connues avant de commencer une chaîne de traitement sont les services disponibles, leurs paramètres, et les états potentiels des documents, des ressources XML et du système (cf. section F.2). L'apprenant ne connaît ni la forme, ni le contenu des documents à l'avance, et il ne sait pas si une extraction sera possible. Ses décisions sont prises dans l'incertain. Pour atteindre notre objectif, nous appliquons donc les techniques de l'apprentissage par renforcement (Reinforcement Learning, RL) pour les processus de décision markoviens (Markov Decision Processes, MDP (Puterman, 1994)). Nous en rappelons les grands principes dans la suite ; pour une introduction détaillée, nous renvoyons le lecteur à [START_REF] Sutton | Reinforcement learning: An introduction[END_REF].

En RL, l'apprenant reçoit une récompense, basée sur les résultats des actions qu'il a choisies. Plus les résultats sont proches des objectifs, plus la récompense est élevée. Le système essaie de maximiser ces récompenses, typiquement en exploitant ce qu'il connaît déjà pour continuer à recevoir de bonnes récompenses, et en explorant de nouvelles actions avec l'espoir d'obtenir des récompenses encore plus importantes.

Le RL est généralement formalisé pour des MDP. Un tel processus modélise l'environnement en termes d'états, dans lesquels des actions, qui mènent à d'autres états du système de manière stochastique, sont possibles. Le fait que l'environnement soit dans un état donné à un certain instant apporte une récompense immédiate à l'agent. L'objectif d'un apprenant est de choisir ses actions de façon à maximiser son espérance de récompenses cumulées, sans connaître, initialement, ni les distributions sur les états résultant d'une action, ni les récompenses associées aux états.

F.3. APPRENTISSAGE PAR RENFORCEMENT

209

Definition 15 (MDP) Un processus de décision markovien est un 5-uplet (S, A, P, R, γ), avec • S un ensemble (ici fini) d'états possibles de l'environnement,

• A un ensemble (ici fini) d'actions (que l'agent peut effectuer),

• P un ensemble de distributions {P a (s, •) | s ∈ S, a ∈ A} ; P a (s, s ′ ) est la probabilité que l'environnement soit dans l'état s ′ après que l'agent a effectué l'action a dans l'état s,

• R une fonction de récompense, que nous supposons définie sur les états ; R(s) est la récompense obtenue par l'agent pour se trouver dans l'état s,

• γ est un facteur d'atténuation dans [0, 1], qui contrôle l'importance des récompenses espérées dans le futur, relativement aux récompenses espérées dans l'immédiat.

Dans le cadre du RL, l'agent (apprenant) ne connaît initialement que les espaces d'états S et d'actions A, ainsi que le facteur γ. À tout instant t, il connaît l'état courant s t de l'environnement, et choisit une action a t . L'environnement passe dans un état s t+1 tiré selon la distribution P at (s t , •), et l'agent est informé de l'état s t+1 et de la récompense r t+1 = R(s t+1 ). Le processus continue en s t+1 . L'agent doit, au fil de ses interactions avec l'environnement, apprendre une série de politiques π 0 , π 1 , . . . , π t , . . . , une politique π t : S → A donnant, pour l'instant t, l'action π t (s) à effectuer si l'état courant est s t . Son objectif est à tout instant de maximiser l'espérance de récompense cumulée, c'est-à-dire l'espérance de la quantité ∞ t ′ =t γ t ′ R(s t ′ ). Ce cadre générique admet de nombreuses variantes (pour un aperçu récent, voir [START_REF] Szepesvári | Algorithms for reinforcement learning[END_REF]).

De nombreux algorithmes ont été proposés dans la littérature pour les problèmes de RL. Dans ce document, nous utilisons d'abord une approche standard, le Q-learning [START_REF] Watkins | Learning From Delayed Rewards[END_REF], avec des récompenses numériques, puis, pour le cadre des récompenses qualitatives, que les approches standards ne traitent pas, nous utilisons l'approche SSB Q-learning [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF]. Notons que notre contribution consiste d'abord à modéliser le problème de l'amélioration continue d'une chaîne de traitement comme un problème de RL, et que d'autres algorithmes pourraient être utilisés.

Pour que ce document soit autosuffisant, nous présentons dans la suite les deux algorithmes de façon générale, mais nous encourageons le lecteur à consulter les articles originaux. Nous avons aussi lancé des tests en utilisant RMax [START_REF] Brafman | R-max-a general polynomial time algorithm for near-optimal reinforcement learning[END_REF] et VMax [START_REF] Rao | V-MAX: A General Polynomial Time Algorithm for Probably Approximately Correct Reinforcement Learning[END_REF], qui sont conceptuellement différents (ils apprennent le MDP sousjacent, au lieu d'apprendre directement la politique), mais leurs temps de convergence et de calcul les rendent prohibitifs pour notre problématique.

F.3.1 Q-learning

Le Q-learning est un algorithme simple, facile à mettre en oeuvre et à paramétrer. Il maintient, pour chaque couple état/action (s, a), une valeur notée Q(s, a), qui représente l'estimation courante, par l'agent, de l'espérance de récompense s'il se trouve dans s, exécute a, puis suit une politique optimale. Lorsque l'agent est dans l'état s t , choisit l'action a t , se retrouve en s t+1 et reçoit une récompense r t+1 , il met à jour son estimation de la valeur Q(s t , a t ) comme indiqué dans l'algorithme 10, où α, le taux d'apprentissage, est un coefficient dans [0, 1] qui fixe l'importance de la dernière expérience (r t+1 + γ max(. . . )) par rapport à l'expérience déjà accumulée (l'ancienne valeur de Q(s t , a t )).

Pour l'exploration, nous utilisons une stratégie ǫ-gloutonne (EG). Le taux d'exploration ǫ ∈ [0, 1] règle le dilemme exploitation/exploration de la manière suivante. À chaque pas de temps t, avec une probabilité ǫ, une action est choisie aléatoirement (l'agent explore) ; sinon, l'agent exploite et choisit simplement l'action a t qui maximise Q(s t , a t ). La résolution de problèmes de RL avec ce type d'information préférentielle/ordinale est le problème étudié par les travaux autour du preference-based RL [START_REF] Akrour | APRIL: Active preference learning-based reinforcement learning[END_REF][START_REF] Fürnkranz | Preference-based reinforcement learning: a formal framework and a policy iteration algorithm[END_REF][START_REF] Wilson | A bayesian approach for policy learning from trajectory preference queries[END_REF]Wirth and Fürnkranz, 2013a;[START_REF] Wirth | Model-free preferencebased reinforcement learning[END_REF]. Dans ce contexte, on ne peut pas maximiser directement l'espérance de récompense cumulée, car les valeurs numériques des récompenses ne sont pas accessibles (tout du moins directement).

Soit r t+1 = R(s t+1 ) 5 Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )(r t+1 + γmax b { Qt (s t+1 , b)} -Qt (s t , a t )) F.3.2 SSB Q-learning
Une approche consiste à calculer une fonction de récompense numérique à partir des retours qualitatifs reçus, mais ceci a typiquement l'inconvénient d'introduire des informations additionnelles, non exprimées par l'utilisateur. Une approche alternative consiste à utiliser des critères de décision adaptés à l'information ordinale. Un critère de ce type est la dominance probabiliste [START_REF] Busa-Fekete | Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm[END_REF], qui cherche à maximiser la probabilité d'obtenir un résultat (un état final f i ) préféré. Plus formellement, soient π 1 , π 2 deux politiques, et soient F 1 , F 2 deux variables aléatoires sur (f 1 , . . . , f k ), où P(F i = f j ) est donné par la probabilité que π i aboutisse au résultat f j ; F x F y indique donc que le résultat obtenu par π x est préféré (ou équivalent) à celui obtenu par π y . Alors le critère de dominance probabiliste s'écrit :

π 1 π 2 ⇔ P(F 1 F 2 ) ≥ P(F 2 F 1 )
En français, une politique π 1 est préférée à une politique π 2 si la probabilité que π 1 obtienne un meilleur résultat que π 2 est plus grande que l'inverse.

En fait, la dominance probabiliste est un cas particulier d'une large classe de critères de décision, appelés Skew Symmetric Bilinear -SSB -utility functions [START_REF] Fishburn | SSB utility theory: an economic perspective[END_REF]. Étant donné φ une fonction d'utilité SSB, φ(π, π ′ ) > 0 (resp. φ(π, π ′ ) < 0) exprime la préférence de l'utilisateur pour π sur π ′ (resp. pour π ′ sur π), tandis que φ(π, π ′ ) = 0 exprime son indifférence. La dominance probabiliste est le cas où φ est toujours 1, 0 ou -1, et ne requiert donc de l'utilisateur que des retours purement ordinaux (« préféré », « indifférent ») sur les états finaux.

Pour traiter des problèmes de RL avec ce critère d'optimisation, nous utilisons l'algorithme SSB Q-learning récemment proposé [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF] (Sutton and Barto, 1998, section 7). Dans ces versions, un paramètre λ ∈ [0, 1] permet et contrôle la rétropropagation des nouvelles expériences le long de la trace de décisions prises : λ = 0 correspond aux algorithmes tels que décrits ci-dessus, tandis qu'à l'autre extrême, λ = 1 correspond aux algorithmes de type Monte-Carlo. Nous définissons les états en utilisant les métadonnées du document source, de la ressource et du système à un instant donné, par exemple, les informations déjà extraites, le temps déjà passé sur ce document, etc. (section F.5). Le système a ainsi une perception de la tâche sous la forme d'états du processus, et chaque passage par un service modifie l'état courant.

Par ailleurs, le système dispose d'un certain nombre d'actions qu'il peut appliquer dans l'état courant : ces actions correspondent au service suivant à lancer, à l'arrêt du traitement et l'enregistrement des événements extraits en base de données, ou à la configuration d'un service. La répétition d'une même action sur un même document est techniquement autorisée, mais sera pénalisée par le système de récompense si elle induit un temps de traitement trop long sans amélioration de l'extraction. La chaîne de traitement n'est plus figée, mais contrôlée par un algorithme de RL.

Les actions font transiter le système d'un état à l'autre. La stochasticité nous permet de prendre en compte le fait que des actions, menées dans un contexte apparemment identique, peuvent ne pas produire le même résultat. À titre d'exemple, choisir de détecter la langue peut résulter en l'extraction de langues différentes, ce qui est pris en compte directement par les actions stochastiques des MDP. Par exemple, supposons que 70 % des documents sources déjà traités étaient en français. L'agent percevra alors une probabilité 0.7 que, lorsqu'il applique l'action consistant à détecter la langue dans un état s t , il arrive dans l'état s fr t+1 , avec s fr t+1 égal à s t mais augmenté de l'information « langue extraite » et de l'annotation « fr ». Plus précisément, les états que perçoit le système sont des états combinatoires, formés par les valeurs d'un certain nombre de descripteurs des documents. Ces états permettent une généralisation en apprentissage, par exemple, nous avons déjà vu que le sujet du document (politique vs. mariage) influence de manière forte l'utilité du mot « alliance » pour l'extraction de l'information. Imaginons qu'un utilisateur cherche des accords entre des pays. On peut espérer que le système apprenne à modifier la chaîne ainsi au fil des interactions :

• si l'état courant a la valeur « vrai » pour le descripteur « sujetExtrait » et la valeur « mariage » pour le descripteur « sujetDuDocument », alors la meilleure action à effectuer consiste à arrêter le traitement (inutile de continuer) ;

• si le sujet est extrait, mais n'est pas « mariage », la meilleure action consiste à lancer un service d'extraction qui utilise « alliance » parmi les mots déclencheurs ;

• sinon, la meilleure action consiste à lancer un service de reconnaissance du sujet du document.

Enfin, les récompenses sont construites à partir du feedback de l'utilisateur de trois manières. La première consiste en une valeur numérique, basée sur les corrections apportées à l'extraction par l'analyste (le cas échéant) et sur le temps passé à traiter le document (section F.7). La seconde consiste en une valeur qualitative, basée sur les préférences purement ordinales de l'utilisateur sur les résultats finaux de l'extraction, et la troisième, qui forme un entre-deux, consiste en une préférence ordinale pondérée (section F.8). Les récompenses r(s) sont donc données à l'agent uniquement pour les états finaux des traitements.

En pratique, il s'agit d'utiliser le fait que les analystes consultent les synthèses produites par le système, synthèses qui présentent les événements extraits, avec des liens vers les documents sources. Les actions de l'analyste, consistant par exemple à corriger l'information extraite, ou encore à consulter une synthèse sans rien corriger, fournissent indirectement un retour sur la qualité du traitement opéré, tout en utilisant, de manière non-intrusive, un travail effectué par les analystes indépendamment de la problématique d'amélioration du processus de traitement.

F.5. CADRE EXPÉRIMENTAL

215

F.5 Cadre expérimental

Nous nous basons sur une chaîne simple mais typique (cf. section F.2), à laquelle nous avons ajouté la possibilité d'utiliser un service en réalité inutile, Geo. La chaîne est écrite comme une route [START_REF] Camel | Apache Camel[END_REF] en XML, où chaque service est défini comme un endpoint. Nous utilisons le Dynamic Router pour donner à BIMBO le contrôle sur les services appelés, leur ordonnancement, et leurs paramètres (ces derniers sont plus spécifiquement le choix des gazetteers de GATE [START_REF] Cunningham | Developing Language Processing Components with GATE Version 8 (a User Guide)[END_REF] pour la détection des événements dans un texte). Notons que les services sont des « boîtes noires » pour BIMBO, et que la connaissance de leurs WSDL, par exemple, n'est pas nécessaire1 .

Les actions disponibles consistent donc à :

• choisir le prochain service parmi {Tika, NGramJ, GATE, Geo} ;

• arrêter le traitement du document (« STOP ») ;

• choisir la configuration du service GATE, c'est-à-dire, ses gazetteers.

Un état est représenté par une affectation de caractéristiques :

• la langue du document, language ∈ {"en"," "} (où " " code « nonextraite ») ;

• le format du document, format ∈ {"text/plain"," "} (où " " code « nonextrait ») ;

• le nombre de secondes écoulées depuis le début du traitement du document courant (arrondi à la dizaine de secondes), durée ∈ R ;

• le nombre de services déjà appliqués sur ce document (groupés pour éviter l'explosion du nombre des états), nbServices ∈ {0 -5, 6 -20, 21+} ;

• si un événement « intéressant » a déjà été extrait, interesting ∈ {true, false} ;

• si un événement quelconque a déjà été extrait, any ∈ {true, false} ;

• le gazetteer de noms choisi (si un tel choix a déjà été fait), nounGazetteer ;

• le gazetteer de verbes choisi (si un tel choix a déjà été fait), verbGazetteer.

Dans nos expériences, ces ensembles donnent un espace de 8 000 états pour 15 actions, et sont choisis à titre illustratif pour la preuve de notre approche. Un système opérationnel prendrait évidemment en compte de nombreuses autres caractéristiques, services et paramètres (type de documents, liste complète de langues, service de traduction, etc.). Notons que même avec un espace d'états/d'actions ainsi enrichi, le temps de calcul ne serait pas un obstacle avec un algorithme de type (SSB) Q-learning, pour lequel les calculs sont instantanés à chaque pas de décision.

Nous avons considéré un corpus de textes, dont les événements d'intérêt sont déjà connus. Le Global Terrorism Database (GTD) est une base de données open source composée des détails de plus de 125 000 événements terroristes mondiaux de 1970de à 2014de (LaFree (2010))). Nous avons mis les types d'événements du GTD en correspondance avec ceux de WOOKIE ; une chaîne d'extraction parfaite extrairait des documents d 1 . . . d N de notre corpus les événements E 1 , . . . , E N ∈ GTD, respectivement (cf. Exemple 12, Exemple 14). Une telle chaîne n'existe pas en réalité. Nous utilisons donc les résultats d'une chaîne « experte » (construite à la main) comme référence. Nous attendons de notre système qu'il apprenne une chaîne qui s'approche de ce but, en apprenant le bon ordonnancement des services, le bon paramétrage des services, et le fait que certains services (dans notre cas, Geo) et certains gazetteers de GATE ne sont pas utiles.

Comme expliqué dans la section F.3, l'agent (l'IA) n'a initialement connaissance que de l'espace des états et des actions, et donc, dans notre cas, commencera à apprendre sans connaissance a priori des documents ou des besoins de l'analyste. Nous dirons qu'une telle IA est « non-formée ». Une fois que l'IA a traité un certain nombre de documents, elle aura appris ce qu'elle considère être une politique optimale. Nous dirons qu'elle est « formée », et nous pourrons vérifier l'efficacité de la politique apprise en obligeant l'IA à la suivre, c'est-à-dire, en mettant ǫ = α = 0, arrêtant ainsi son exploration et son apprentissage.

Ici, nous testons la capacité d'apprentissage à partir de rien d'une IA sur des documents complètement inconnus. Bien sûr, en production, nous pourrions capitaliser sur l'expertise existante en initialisant l'IA avec une politique basée sur celle d'une chaîne standard. La similarité temporelle σ T (T 1 , T 2 ) est de 1 s'il y a au moins un atome en commun entre les dimensions temporelles de E 1 et E 2 . Sinon, nous utilisons l'information dérivée (jour, mois, année, jour de la semaine). Par exemple, pour T 1 = {7 October 1969} et T 2 = {October}, σ T (T 1 , T 2 ) = 1 10 . Pour T 3 = {T uesday}, l'intersection avec T 1 est vide, mais en notant que le 7 octobre 1969 a été un mardi, nous obtenons σ T (T 1 , T 2 ) = 1 7 . Ces valeurs ont été choisies par expérience, et soulignent la difficulté d'associer une valeur numérique à une comparaison. Cette difficulté sera levée par l'approche basée sur du feedback qualitatif.

Enfin, pour la similarité entre les dimensions agentives de E 1 et E 2 , nous utilisons la distance de Levenshtein sur chaque paire d'agents a 1 , a 2 pris dans A 1 , A 2 (nombre minimal de caractères à supprimer, insérer ou remplacer pour passer d'une chaîne à l'autre), distance rendue « floue » en considérant les sous-séquences de la chaîne principale [START_REF] Ginstrom | National Consortium for the Study of Terrorism and Responses to Terrorism (START)[END_REF], et notée F SLD(a i , a j ). Nous définissons σ A (A 1 , A 2 ) comme (1-max{F SLD(a 1 , a 2 ) | a 1 ∈ A 1 , a 2 ∈ A 2 }), si elle est au-dessus d'un certain seuil θ, et 0 sinon (après expérimentation, θ = 0.45 donne de bons résultats). Par exemple, pour

A 1 = {Jharkhand ✭ ✭ ✭ ✭ ✭ ✭ ❤ ❤ ❤ ❤ ❤ ❤ anti-Maoist ✭ ✭ ✭ ✭ ✭ ❤ ❤ ❤ ❤ ❤ Maobadi ✭ ✭ ✭ ✭ ✭ ❤ ❤ ❤ ❤ ❤ Protirodh ✭ ✭ ✭ ✭ ✭ ✭ ❤ ❤ ❤ ❤ ❤ ❤ Committee member} et A 2 = {Jharkhand ✘ ✘ ✘ ❳ ❳ ❳
Party member}, on obtient une distance Levenshtein floue de 11, donc σ A (A 1 , A 2 ) = 1 -11 22 = 0.5.

Nous ne nous intéressons pas ici à l'association d'entités telles que François Hollande / le président, mais la modularité du système permettrait de prendre en compte cette similarité facilement, en s'appuyant sur des ressources adéquates.

Les volumes de documents traités en production (idéalement, tous les documents du web) étant très importants, le temps passé à traiter un document doit être minimisé. La qualité des extractions prend donc en compte la similarité entre l'événement cible et l'événement extrait (le cas échéant), mais aussi le temps de traitement. Précisément, en notant σ(E e , E g ) la similarité moyenne entre les événements extraits (s'ils existent) et l'événement du GTD, et t le temps passé par la chaîne sur le document, nous définissons la qualité de l'extraction Q par σ(E e , E g )/t pour σ(E e , E g ) = 0, et par Q = -t sinon (c'est-à-dire s'il n'y pas d'extraction, ou une similarité nulle avec l'événement cible). Nous formalisons ainsi le fait que l'extraction d'événements corrects est primordiale, doit être effectuée en un temps raisonnable, et que l'agent doit détecter rapidement, le cas échéant, qu'il n'y a pas d'événement intéressant à extraire.

F.7 Tests avec un feedback numérique

Durant ces tests, l'IA reçoit une récompense numérique sur la qualité Q de ses résultats, telle que définie dans la section F.6. En production, ceci équivaut à la récompenser en fonction de l'événement qu'elle a extrait (si elle en a extrait un) et de l'événement tel que corrigé par l'analyste humain, ce dernier étant considéré comme l'événement « parfait » pouvant être extrait du document (si un événement peut être extrait). Ceci est réalisé de manière non-intrusive, dans la mesure où cela repose sur des corrections que l'analyste a besoin d'effectuer de toute façon, mais nécessite un réglage fin (2) Nous avons testé à la fois la politique apprise par cette IA formée, et une IA non-formée sur 1 000 nouveaux documents choisis aléatoirement depuis la GTD. Seulement 29% d'entre eux contenait des événements Bombing « intéressants », 7% d'autres événements, et 64% aucun événement extractible. Nous avons réglé la chaîne « experte » (c.f. la section F.5) pour n'extraire que des événements Bombing.

L'IA non-formée (partant de zéro) est parvenue à extraire 39% des événements possible des 1 000 documents inconnus, et nous voyons sur la Figure F.4 qu'elle généralise bien, sa performance s'améliorant au fur et à mesure des documents rencontrés.

L'IA formée a démontré une politique similaire à celle de la chaîne experte, en extrayant 100% des événements. Ainsi, elle aussi généralise bien, appliquant avec succès la politique apprise à des documents inconnus. Notons que les IAs ont également appris à ordonner la chaîne, e.g., dans l'état F.7. TESTS AVEC UN FEEDBACK NUMÉRIQUE 221 correspondant à un document dont ni le format ni la langue ont été détectées, et dont le temps de traitement est quasi-nul, la meilleure action apprise était de passer le document au service Tika. Dans l'état correspondant à un document où au moins un événement est extrait, quelle que soit la valeur des autres attributs, l'IA arrête le traitement de ce document.

Comme espéré, l'IA a aussi appris à optimiser la chaîne. Elle n'a pas utilisé le service inutile (Geo), et a seulement utilisé la liste de verbes bombing. Cependant, elle a de manière surprenante préféré les noms injure à ceux de bombing. Après enquête, nous avons trouvé que le service GATE fourni n'utilise que les verbes pour l'extraction d'événements, ignorant les noms. Ceci montre que l'IA est capable de découvrir des stratégies qui ne sont pas évidentes pour l'expert ayant calibré la chaîne. Parallèlement, après le traitement de chaque 100 documents, nous avons testé une IA initialisée avec la politique apprise à ce point, et les valeurs de ǫ et α à 0, sur 10 documents inconnus contenant tous des événements extractibles.

Nous voyons dans les Figures F.7-F.9 que la première extraction n'a lieu qu'à 173 documents, et que les extractions sur les premiers 700 documents ont une qualité inférieure à celle de la chaîne « experte ». Au document 701, en revanche, l'IA commence à recevoir des récompenses similaires à celles de la chaîne « experte », ce qui indique que la qualité de ses extractions s'améliore. Enfin, les analystes ne sont pas des entraîneurs dédiés, et ne corrigeront pas toutes les extractions. Nous avons donc testé une récompense donnée à la fin de chaque lot, avec une probabilité de 10 % sur chaque extraction (sinon, l'IA ne recevait aucune récompense pour cette extraction).

Même avec une récompense aussi sporadique, l'IA est parvenue à extraire 50 % des événements possibles après seulement 1 890 documents (30 lots), et 100 % après 5 103 documents (81 lots). La qualité de ces deux politiques est représentée sur les Figures F.10c,F.10d, et suggère que la capacité à extraire des événements et à extraire des événements corrects croissent conjointement. Notons que la qualité la plus basse (en comparaison avec la chaîne experte) que l'on observe dans la dernière courbe est due au temps de traitement, et non à la similarité avec les événements attendus.

F.8 Tests avec un feedback intuitif

Les tests présentés en section F.7 supposent que l'utilisateur est capable de donner un feedback numérique à l'agent, ce qui n'est pas réaliste. Nous présentons ici donc une évaluation expérimentale de notre méthode en utilisant un feedback qualitatif. Rappelons que l'approche n'a besoin que d'un retour sur les qualités relatives de deux extractions ; la comparaison entre politiques se fait alors sur le critère de dominance pondérée, relativement à ces comparaisons.

Nous explorons ici deux types de comparaison entre les résultats de deux traitements f, f ′ . Le protocole expétimental est similaire à celui de la section F.7, et nous prenons comme références la chaîne experte, ainsi que les résultats obtenus par le Q-learning avec une récompense numérique (noté QL dans cette section).

Premièrement nous encourageons l'IA à d'abord extraire les événements, et à le faire rapidement, ou à vite reconnaître qu'il n'y a aucun événement à extraire. Ceci repose sur la corrélation entre la capacité à extraire n'importe quel événement et celle à extraire des événements corrects, tel que démontré à la section F.7 pour le Q-learning. En l'état, notons que cette récompense peut même être calculé automatiquement, sans intervention de l'analyste. Évidemment, cette relation de préférence pourrait être complétée avec des informations supplémentaires sur les résultats obtenus, tels que la qualité perçue de l'extraction, etc. Cette récompense noté DOM est défini comme :

f ≻ DOM f ′ ⇔ φ DOM (f, f ′ ) = 1 ssi (i) le traitement f a extrait un événement mais pas f ′ , ou (ii) aucun ou bien les deux ont extrait un événement, et f a été plus rapide ; f ∼ DOM f ′ ⇔ φ DOM (f, f ′ ) = 0 ssi les deux ont pris à peu près le même temps, i.e., l'écart entre les deux temps de traitement est inférieur à margin (réglé à 5) secondes.

Nous notons MAG le deuxième type de récompense. Intuitivement, MAG est un intermédiaire entre QL et DOM, qui peut être vu comme une forme pondérée de dominance probabiliste, demeurant toutefois naturel. Nous ac-cordons plus d'importance dans l'ordre à : l'extraction des événements, puis l'extraction des événements exactes, et enfin la vitesse de traitement : φ MAG (f, f ′ ) = 1000 ssi f a extrait un événement et pas f ′ ; φ MAG (f, f ′ ) = 100 ssi les deux ont extrait un événement, mais l'extraction de f est de meilleure qualité (section F.6) que celle de f ′ ; φ MAG (f, f ′ ) = 10 ssi les extractions étaient de qualité similaire, ou aucune extraction a eu lieu, mais f a été plus rapide par au moins margin secondes ; φ MAG (f, f ′ ) = 0 ssi f et f ′ ne tombent dans aucun des cas précédents. Les valeurs 0, 10, 100, 1000 reflètent une différence de magnitude entre l'importance des différents critères, sans besoin de les régler plus précisément.

Nous avons réalisé un jeu de tests étendu avec QL, DOM et MAG, en faisant varier les paramètres γ, ǫ, et λ (voir la section F.3). Nous avons exécuté les IA non-formées sur un ensemble de 5 000 documents GTD (présentés dans le même ordre, et rencontrés une seule fois dans tous les tests). Les actions étaient choisies parmi dix gazetteers, les services Tika, NGramJ, GATE, Geo, et STOP. La qualité du traitement de chaque document a été mesurée comme décrit en section F.5 pour toutes les approches. Nous insistons sur le fait que la qualité est donc mesurée avec le même critère (numérique) que celui utilisé pour donner un récompense à QL, et ceci pour les trois approches ; DOM et MAG reçoivent donc un feedback à première vue peu informatif vis-à-vis du critère à optimiser, demandant donc un effort cognitif faible aux experts.

Évidemment, comme les IA apprenaient de zéro, la politique initiale était de mauvaise qualité. Nous nous sommes intéressés au temps requis (en termes de nombre de documents traités) pour atteindre une bonne politique.

Le paramètre α a été réglé comme dans [START_REF] Gilbert | Model-free reinforcement learning with skew-symmetric bilinear utilities[END_REF], i.e. diminuant avec le nombre de visites de la paire état / action courante. Nous avons exécuté 36 combinaisons de tests en faisant varier les autres paramètres, afin d'obtenir un ensemble de résultats étendu, et de mesurer la robustesse visà-vis du choix des paramètres : -les algorithmes QL, DOM, et MAG, -le paramètre EG ǫ divisé par 2 après 2500 ou 1000 documents, γ = 0.9 et γ = 1, λ = 0, λ = 0.95, et λ = 1.0. La Figure F.11 montre les courbes les plus intéressantes et représentatives. Nous avons tracé uniquement la qualité d'extraction lorsque l'IA a suivi sa meilleure politique pendant tout le traitement (elle a exploité) avec la ligne bleu en pointillés, et celle de la chaîne « experte » (ligne continue rouge)2 . Pour une meilleure lisibilité, nous avons lissé les courbes en prenant les moyennes sur les ensembles de 50 documents avec le même ordre de traitement.

QL a donné d'excellents résultats, mais a montré une certain sensibilité au changement de paramètres. γ = 0.9 donne d'excellents résultats avec λ = 0.95 (Figure F.11a), où l'IA apprend une politique « assez bonne » après seulement 250 documents (les événements sont correctement extraits, mais elle prend quelques secondes de plus que la chaîne « experte »), et une politique optimale après 1 200 documents. γ = 0.9 donne aussi de très bons résultats avec λ = 0 (non montré). Cependant, avec γ = 0.9 et λ = 1, les résultats sont médiocres, et γ = 1 (Figure F.11b) donne de très mauvais résultats indépendamment de λ : l'IA a appris à s'arrêter très tôt, qui suggère une aversion au risque.

DOM, comme QL, a été sensible au choix de γ et λ. Avec γ = 0.9 et λ = 1 (non montré) et λ = 0.95 (Figure F.11f) les résultats sont très mauvais. Toutefois des résultats acceptables (non montré) ont été obtenus avec γ = 0.9 et λ = 0 pour la stratégie de réduction de ǫ la plus longue, et avec γ = 0.9 et λ = 0 ou λ = 0.95 pour la stratégie la plus courte. Avec γ = 1, les résultats allaient de bons à excellents, et la Figure F.11e montre les meilleurs résultats pour γ = 1, λ = 1, où une politique « assez bonne » est apprise après 1 000 documents et se stabilise à l'optimum après 3 750 documents (vérifié jusqu'à 10 000 documents -non montré).

MAG s'est révélé robuste au changement de paramètres, apprenant rapidement une politique au moins « assez bonne » dans tout les cas (e.g. la Figure F.11c).

Nous avons remarqué que le SSB Q-Learning (DOM ou MAG) apprend parfois une politique optimale mais que la performance se dégrade. Même si les événements sont toujours bien extraits, le traitement devient trop lent (voir les Figures F.11g,F.11h) : l'IA apprend qu'un appel à GATE depuis un état donné produit une bonne récompense, et si γ et λ ne sont pas correctement réglés, bien que cela prenne plus de temps, elle commence à préférer cette action à l'arrêt.

Enfin, nous avons constaté une légère amélioration des résultats avec une réduction plus rapide de ǫ (tous les 1 000 documents vs tout les 2 500), c'est-à-dire, avec globalement moins d'exploration.

En F.9 Discussion et perspectives

Revenons à notre usine de voitures étrange, et observons comment ils ont réussi.

Elle a recruté une BIMBO comme chef de département, chargée de s'occuper des retours utilisateur. À son tour, celle-ci a recruté une équipe de choc d'IA. Chaque IA a été munie des plans de tous les modèles de voiture possibles. Ensuite, elle a commencé à organiser l'usine.

Les IA étaient rémunérés aux résultats, avec des primes pour des clients satisfaits. Afin de mesurer cette satisfaction client, ils ont émis un sondage à chaque fois qu'une voiture a été achetée. Quelques clients ne rendaient pas immédiatement leur formulaire, et la vaste majorité de clients ne l'ont jamais rempli. Cependant, les IA ont réussi à bien gagner leur vie, et en général, la satisfaction client a augmenté. Puis, un petit futé à l'usine a suggéré le dispositif de contrôle continu, qui pourrait fonctionner en parallèle des sondages. Les IA étaient ravies parce qu'elles étaient toujours payées, même si les clients ne voulaient pas remplir un formulaire. Les clients étaient enthousiastes également. Ils n'avaient plus besoin de comparer la menthe et l'olive, ou l'asperge et l'avocat, ils pouvaient tout simplement dire « j'aime bien le vert, de préférence foncé ». BIMBO s'est rendu compte que les IA, parfois, étaient un peu lentes à remplir la paperasse. Elle a donc recruté une assistante personnelle, Dora, qui les a aidées. L'usine a connu un tel succès, qu'elle a diversifié ses activités en achetant une petite société d'analyse d'images.

F.9.1 Conclusion

Nous avons modélisé une chaîne de traitement de documents sous forme de processus de décision markovien, que nous avons résolu en utilisant l'apprentissage par renforcement. Pour implémenter cela, nous avons développé l'application modulaire BIMBO (Benefiting from Intelligent and Measurable Behaviour Optimisation) dans laquelle nous pouvons « brancher » différents algorithmes d'apprentissage par renforcement, services web et modèles afin de mesurer leur impact sur l'apprentissage.

Nous avons établi que notre approche donne de bons résultats avec un feedback numérique sporadique. Nous avons ensuite intégré une fonction de récompense formalisant des préférences utilisateur exprimées naturellement, permettant d'obtenir d'aussi bons résultats, tout en exigeant moins d'efforts cognitifs pour définir le feedback. Nous avons ainsi montré qu'il est possible d'aboutir à une chaîne de traitement capable de s'améliorer sans intervention ou réglage par un utilisateur humain, et qui obtient son feedback de façon non-intrusive.

Notre approche peut être utilisée pour d'autres traitements, par exemple la reconnaissance des objets dans une image. En effet, plusieurs types de capteurs et de nombreux algorithmes existent pour cela, et il n'est pas encore évident de les combiner optimalement. Nous avons appliqué notre approche, en utilisant la plateforme BIMBO et une variation de Q-Learning(λ), Dora [START_REF] Nicart | Dora Qlearning -making better use of explorations[END_REF], aux tâches de segmentation (le pré-traitement de l'image) et de la ROC (reconnaissance optique de caractères). Notre but est la détection des éléments d'intérêt, pour suggérer automatiquement des titres depuis les pages de garde des documents. Spécifiquement, nous avons utilisé les caractéristiques des pages, leurs images, et les paramètres utilisés par les services pour définir les états du MDP. Les actions consistent à choisir les valeurs de ces paramètres, ou à envoyer l'image au traitement. Nous avons constaté des résultats très encourageants avec un espace de plus d'un million d'états.

F.9.2 Perspectives

Dans ce document, les ensembles d'états et d'actions ont été choisis pour démontrer la validité de notre approche. La prochaine étape consiste à produire des chaînes plus complexes, en ajoutant des services alternatifs (par exemple, la traduction), en étendant l'ensemble d'états (par exemple avec la liste complète de langues), et en introduisant une gamme de documents d'entrée plus large. Notre objectif est de montrer que le système est capable de construire

  quality of the results by document for γ = 0.9; varying λ; slow reduction of ǫ = 0.4/2 every 2 500 documents, minimum 0.05. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 The quality of the results by document for γ = 1; varying λ; slow reduction of ǫ = 0.4/2 every 2 500 documents, minimum 0.05. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 The quality of the results by document for γ = 0.9; varying λ; faster reduction of ǫ = 0.4/2 every 1 000 documents, minimum 0.05. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 The quality of the results by document for γ = 1; varying λ; faster reduction of ǫ = 0.4/2 every 1 000 documents, minimum 0.05. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5 Results for MAG with φ in -30, -20, -10, 0, 10, 20 and 30. . . 10.6 Results for MAG with φ in -300, -200, -100, 0, 100, 200 and 300. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7 Results for MAG with φ in -100 000, -1000, -10, 0, 10, 1000 and 100 000. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8 Results for MAG with φ in -1000, -100, -10, 0, 10, 100 and 1000.116 12.1 Sutton and Barto (1998)'s forward (theoretical) view: looking

  Figure 1.1 -An example of a possible treatment chain which extracts information from a video. The faces are identified and tagged with names. In parallel, the languages spoken are identified and the voice recording is transcribed. The native language extractor is used to summarise the information contained in the transcript, whilst the transcription is also translated. The polarity of the opinions expressed, and their subjectivity is assessed, and an English extractor used. In all cases the information is indexed in a database.

  This part of the manuscript represents the bulk of the work done during the three years of the thesis.

Figure 4

 4 Figure 4.2 -A simple but typical Weblab chain: a document, d, is converted into an XML resource, r 0 , and annotations, u i , are added by each service. Finally the results are stored in a database for user consultation.

  4.2. WHO ARE THE ANALYSTS?27 <r e s o u r c e t y p e=" Document " u r i=" w e b l a b : a a a "> <a n n o t a t i o n u r i=" w e b l a b : a a a#a0 "> <w p : o r i g i n a l C o n t e n t r e s o u r c e=" f i l e : w e b l a b . c o n t e n t " /> < w p : o r i g i n a l F i l e S i z e>255</ w p : o r i g i n a l F i l e S i z e> < d c : s o u r c e>documents / e v e n t . t x t</ d c : s o u r c e> <w p : o r i g i n a l F i l e N a m e>e v e n t . t x t</ w p : o r i g i n a l F i l e N a m e> <d c : m o d i f i e d>2015-02-14 T 1 9 : 5 2 : 2 1 +0100</ d c : m o d i f i e d> < w p : c o l l e c t e d>2015-02-10 T 0 0 : 1 1 : 0 0 +0200</ w p : c o l l e c t e d> </ a n n o t a t i o n> <a n n o t a t i o n u r i=" w e b l a b : a a a#a1 "> <w p : i s P r o d u c e d B y r e s o u r c e=" w e b l a b : t i k a " /> <d c : f o r m a t>t e x t / p l a i n</ d c : f o r m a t> </ a n n o t a t i o n> <a n n o t a t i o n u r i=" w e b l a b : a a a#a2 "> <w p : i s P r o d u c e d B y r e s o u r c e=" w e b l a b: n g r a m j " /> <d c : l a n g u a g e>en</ d c : l a n g u a g e> </ a n n o t a t i o n> <m e d i aU n i t t y p e=" w l : T e x t " u r i=" w e b l a b : a a a#0"> <c o n t e n t>4 / 2 9 / 1 9 7 1 : I n a s e r i e s o f two i n c i d e n t s t h a t might have been p a r t o f a m u l t i p l e a t t a c k , s u s p e c t e d members o f t h e Chicano L i b e r a t i o n F r o n t bombed a Bank o f America b r a n c h i n Los A n g e l e s , C a l i f o r n i a , US . There were no c a s u a l t i e s but t h e b u i l d i n g s u s t a i n e d $1 600 i n damages .</ c o n t e n t> </ m e d i a U n i t> </ r e s o u r c e>

Figure 4

 4 Figure 4.3 -Simplified XML of a WebLab resource mid-chain, showing annotations added by Tika and NGramJ.

CHAPTER 4 .

 4 THE INDUSTRIAL CHALLENGE ure 4.4, for example, we see the named entities highlighted, and the user adding an organisation dublincore (the edit and delete screens are similar, not shown).

  Figure 4.5 shows an example of a tooltip, allowing the user to access any extra information that is stored in the database and connected to that entity by hovering over it.

Figure 4

 4 Figure 4.4 -Adding the organisation dublincore using the open-source interface.

Figure 4

 4 Figure 4.5 -An example of a tooltip giving further information about an entity using the open-source interface.

PP(Figure 5 . 3 -

 53 Figure 5.2 -Shopping for grapes and milk -first week.

Figure 7 . 1 -

 71 Figure 7.1 -The modules of BIMBO. The Dynamic Router defines the available services, the controller holds the stop-watch and controls the flow of input documents, etc.

Figure 7

 7 Figure7.3 -The log from the treatment of a single document roughly halfway through the learning process. The numbers on the left of each line in the log show the time in nanoseconds (we have taken out the days / hours / minutes to aid readability).

  of the corpusTo test our approach in the industrial context of the CIFRE contract, we used an open-source database in which the events are already known: the Global Terrorism Database (GTD) (Global Terrorism Database, 2016b; La-Free, 2010), consisting of details of over 125 000 worldwide terrorist events from 1970 to 2014.Each of these events lists up to three sources. We initially extracted the urls and retrieved the web pages (see Figure7.4 for an example).

Figure 7

 7 Figure 7.4 -Original document (web page) -shortened (see Appendix D for the full page. This web page is reproduced here with the kind permission of Kasturi & Sons Ltd (KSL).

CHAPTER 7 .

 7 BIMBO: A FLEXIBLE PLATFORM One killed, another abducted in West Bengal Attacks occur on the first day of the two-day shutdown Midnapore/Ranchi/Patna: Maoists on Monday unleashed a wave of attacks in which a Jharkhand Party member was killed in West Bengal. E = (C, T, G, A) with C = {DeathEvent}, T = {Monday}, G = {West Bengal}, and A = {Jharkhand Party member}.

Figure 7 . 5 -

 75 Figure 7.5 -Part of the text from the web page in Figure 7.4 and the corresponding event dimensions extracted using WebLab.

Figure 7 . 6 -

 76 Figure 7.6 -The event summary in the GTD corresponding to Figure 7.4 and the corresponding formalised event.

  Figure 7.7 -Similarity of 0.66075 between the extracted event, and the corresponding event in the GTD: Semantic: 1; Temporal: 0.143; Spatial: 1.0; Agentive: 0.5; (a = b = c = d = 1).

  Figure 8.1 -The rewards received by the untrained AI, the trained AI and the expert chain over 1000 unknown documents.

  Figure 8.2 -Percentage of documents containing extractable events by 100 documents for tests with few extractable events.

  Figure 8.3 -Percentage of events extracted using the policy learnt after each 100 documents.

  Figure8.4 -Extraction quality for the expert chain against that of the untrained AI for documents 1 -2 400 with a very low percentage of extractable events.

  Figure 8.7 -Odd behaviour from the AI with full feedback and ǫ divided by 2 every 100 documents to a minimum value of 0.1.

  Figure8.9 -Even worse behaviour from the AI with full feedback and increased exploration as ǫ is divided by 2 every 200 documents.

Figure 8

 8 Figure 8.10 -Full feedback with aggressive non-exploration strategies based on the number of visits to each state.

  Figure8.12 -Full feedback delayed until 100 documents have been treated.

  Figure 8.13 -Partial feedback delayed until 100 documents have been treated.

Figure 9

 9 Figure 9.2 -Two policies, where we have defined the final states in terms of wealth levels, or rewards.

  → v =(time, agentive error [Boolean], spatial error [Boolean], . . . ) or → w=(time, should have extracted something, . . . ) or → x =(time, was right not to extract anything, . . . ) etc.

  Figure 9.3 -A potential partial preference relation over final state criteria.

Figure 9

 9 Figure 9.4 -Numerical rewards received by the AI after training over 30 ×100 documents with immediate feedback, compared with the expert chain.

Figure 10 Figure 10 Figure 10 Figure 10

 10101010 Figure10.1 -The quality of the results by document for γ = 0.9; varying λ; slow reduction of ǫ = 0.4/2 every 2 500 documents, minimum 0.05.

  Table10.1 -A colour-coded easy reference showing the quality of the results for Q-Learning(λ) and SSB Q-Learning(λ) varying the number of documents after which ǫ is divided, and the coefficients γ and λ. Dark red indicates a very poor result, whilst dark green indicates an excellent result. SSB QL(λ): Type of φ divide ǫ every γ 4a Figure 10.4b Figure 10.4c 1 Figure 10.4d Figure 10.4e Figure 10.4f 0.95 Figure 10.4g Figure 10.4h Figure 10.4i

  With γ = 1, regardless of the ǫ reduction strategy, and regardless of λ (Figure 10.2a, Figure 10.2d, Figure 10.2g, Figure 10.4a, Figure 10.4d and Figure 10.4g), the results were very bad: the AI learnt to STOP very early, suggesting a risk-adverse behaviour as γ = 1 puts a heavy emphasis on the future possible rewards.

Figure 10 . 5 -

 105 Figure10.5 -Results for MAG with φ in -30, -20, -10, 0, 10, 20 and 30.

Figure 10 . 6 -

 106 Figure 10.6 -Results for MAG with φ in -300, -200, -100, 0, 100, 200 and 300.

Figure 10 . 7 -

 107 Figure10.7 -Results for MAG with φ in -100 000, -1000, -10, 0, 10, 1000 and 100 000.

Figure 12

 12 Figure 12.1 -Sutton and Barto (1998)'s forward (theoretical) view: looking forward in time to update the current state.

Figure 12

 12 Figure 12.2 -Sutton and Barto (1998)'s backward (mechanistic) view: looking backward in time to update the previous states.

  agent receives reward of 10 and back-propagates it along the trace.

  agent receives the reward of 1000 but cannot back-propagate it.

Figure 12 . 4 -

 124 Figure 12.4 -An example of how QL(λ) cuts the trace on an explore, preventing the back-propagation of useful learnt values.

  receives reward of 10 and back-propagates it along the trace.

  Algorithm 4: Q-learning(λ) Data: MDP M 1 while True do 2 Choose a t in s t using the EG exploration strategy 3 Play a t , observe s t+1 , and let r t+1 = R(s t+1 ) 4 Update Q-Values using Algorithm 5 Algorithm 5: Q-Learning(λ): update Q-Values.

2

  Choose a t in s t using the EG exploration strategy 3 Play a t , observe s t+1 , and let r t+1 = R(s t+1 ) 4 Update Q-Values using Algorithm 8

Figure 12 . 6 -

 126  were run on the same randomly generated MDP of 80 states and 6 actions. Results were measured using the episodic distance averaged over 50 runs.

Figure 12 . 8 -

 128 Figure 12.8 -Dora and QL(λ) were run on the same Long Thin MDP (Figure 12.7) of 15 states and 2 actions. Results were measured using the episodic distance averaged over 50 runs.

  Figure12.9 -Dora, QL(λ) and Naive QL(λ) were run on the same Long Thin MDP (see Figure12.7) of 15 states and 2 actions. Results were measured using the episodic distance averaged over 100 runs.

  Figure12.10 -Dora, QL(λ) and Naive QL(λ) were run on the same randomly generated MDP of 80 states and 6 actions. Results were measured using the episodic distance averaged over 100 runs.

Figure 13 .

 13 Figure 13.1 -An example of a scanned front page, in which the title needs to be detected. Reproduced with the kind permission of Valentin Laforge.

Figure 13 . 2 -

 132 Figure 13.2 -Documents contained noise which could hinder the text recognition software, and so had to be cleaned. Images reproduced with the kind permission of Valentin Laforge.

  (a) The dominant lines in the document were established. (b) The average of the perpendiculars gave the degree of rotation. (c) If the document was upside down, it was impossible to rectify.

Figure 13 .

 13 Figure 13.3 -The document was scanned at a slant and had to be reoriented. Images reproduced with the kind permission of Valentin Laforge.

Figure 13 . 4 -

 134 Figure 13.4 -The document was eroded and dilated until a binary mask showed rectangular regions. Images reproduced with the kind permission of Valentin Laforge.

  143 (a) An example of the front page of a scanned document, cleaned and reoriented, in which the title needs to be detected. (b) Candidates for the title have been detected. The most likely are outlined in green, the least likely in red.

Figure 13 .

 13 Figure 13.5 -Detection of titles by the OCR service. Images reproduced with the kind permission of Valentin Laforge.

Figure 13 . 6 -

 136 Figure 13.6 -The modules of BIMBO applied to the detection of objects in an image.

Figure 13 .

 13 Figure 13.8 -The default parameters as determined by the expert.

Figure 13 . 9 -Figure 13 .

 13913 Figure 13.9 -The quality of the image analysis given ǫ starting at 0.2, divided by 2 every two times through the corpus, and stabilising at 0.85.

  s p r i n g f r a m e w o r k . o r g / schema / b e a n s / s p r i n g -beans -3 . 0 . xsd "> < d e s c r i p t i o n> C o n t a i n s t h e c o n f i g u r a t i o n p a r a m e t e r s f o r BIMBO: B e n e f i t i n g from I n t e l l i g e n t and M e a s u r a b l e B e h a v i o u r O p t i m i s a t i o n </ d e s c r i p t i o n> <bean i d=" p r o p e r t i e s " c l a s s=" o r g . s p r i n g f r a m e w o r k . b e a n s . f a c t o r y . c o n f i g . P r o p e r t y P l a c e h o l d e r C o n f i g u r e r "> <p r o p e r t y name=" i g n o r e R e s o u r c e N o t F o u n d " v a l u e=" t r u e " /> <p r o p e r t y name=" p r o p e r t i e s "> <p r o p s> < !--Your home h e r e --> <prop key=" h o m e D i r e c t o r y ">/home/ n i c a r t /</ prop> < !--The f o l d e r c o n t a i n i n g a l l t h e s t u f f n e c e s s a r y t o run BIMBO --> <prop key=" b u n d l e T e s t i n g ">${ h o m e D i r e c t o r y } B u n d l e T e s t i n g /</ prop> < !--The b u n d l e i n s t a n c e s h o u l d be i n t h e d i r e c t o r y ${ h o m e D i r e c t o r y }/${ b u n d l e T e s t i n g }/ --> <prop key=" b u n d l e I n s t a n c e ">T h e s i s -WebLab</ prop> < !--I f u s i n g t h e b u n d l e p r o v i d e d , t h i s s h o u l d n o t c h a n g e --> <prop key=" s p a r q l E n d p o i n t ">h t t p : // l o c a l h o s t : 3 0 3 0</ prop> < !--Used t o f i n d t h e r o u t e s f i l e , and t h e g a z e t t e e r s which t h e AI u s e s t o r e p l a c e t h o s e i n GATE --> <prop key=" t h e s i s -a i R e s o u r c e s ">${ b u n d l e T e s t i n g } r e s o u r c e s / </ prop> </ p r o p s> </ p r o p e r t y> </ bean> <bean i d=" C o n f i g A I " c l a s s=" c o n f i g . C o n f i g A I "> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e a r t i f i c i a l i n t e l l i g e n c e </ d e s c r i p t i o n> < !--The number o f documents t o be p r o c e s s e d p e r b a t c h --> <p r o p e r t y name=" noDocs " v a l u e=" 1 " /> < !--The number o f b a t c h e s , e . g . s e t t o 5 i f you want t o run t h e same documents t h r o u g h 5 t i m e s --> <p r o p e r t y name=" n o B a t c h e s " v a l u e=" 10 " /> < !--Whether t h e same documents s h o u l d be u s e d e a c h b a t c h --> <p r o p e r t y name=" sameDocs " v a l u e="FALSE" /> < !--Whether v a l u e s have been s t o r e d t h a t can be u s e d t o c o n t i n u e a run a l r e a d y s t a r t e d . --> <p r o p e r t y name="sameRun" v a l u e="FALSE" /> < !--S h o u l d we u s e t h e AI ( t r u e ) , o r t h e s t a n d a r d c h a i n ( f a l s e ) --> <p r o p e r t y name=" useAIForRun " v a l u e="TRUE" /> < !--A l g o r i t h m t o u s e : QL = QLearning , RM = RMax, RMR = RMaxRandom , SSBRM = SSBRMax , VM = VMax VMR = VMaxRandom , SSBQL = SSBQLearning , DORA = Dora --> <p r o p e r t y name=" al go T oU s e " v a l u e="QL" /> </ bean> <bean i d="ConfigCommonQLParams" c l a s s=" c o n f i g . ConfigCommonQLParams"> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e QLearning and SSB QLearning a l g o r i t h m s 165 APPENDIX A. AN EXAMPLE CONFIGURATION </ d e s c r i p t i o n> < !--The d i s c o u n t r a t e t o be u s e d f o r s o l v i n g t h e problem --> <p r

Figure C. 1 -

 1 Figure C.1 -Chain definition in XML (part 1) -declaration of web services.

Figure C. 2 -

 2 Figure C.2 -Chain definition in XML (part 2) -declaration of Camel context and file consumer route.

  Director-General of Police (Headquarters) Neelmani said tight security arrangements were in place to counter the Maoist bandh. -PTI Printer friendly page Send this article to Friends by E-Mail Front Page News: ePaper | Front Page | National | Tamil Nadu | Andhra Pradesh | Karnataka | Kerala | New Delhi | Other States | International | Opinion | Business | Sport | Miscellaneous | Engagements | Advts: Retail Plus | Classifieds | Jobs | Obituary | Updates: Breaking News | guidelines Pakistani

The

  Hindu Group: Home | About Us | Copyright | Archives | Contacts | Subscription Group Sites: The Hindu | The Hindu ePaper | Business Line | Business Line ePaper | Sportstar | Frontline | Publications | eBooks | Images | Ergo | Home | Copyright © 2009, The Hindu. Republication or redissemination of the contents of this screen are expressly prohibited without the written consent of The Hindu 2 of 2 APPENDIX E. FULL WEBLAB RESOURCE <d c : l a n g u a g e x m l n s : d c=" h t t p : // p u r l . o r g / dc / e l e m e n t s / 1 . 1 / ">en</ d c : l a n g u a g e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#a4 "> <d c t e r m s : c r e a t e d x m l n s : d c t e r m s=" h t t p : // p u r l . o r g / dc / t e r m s / "> 2014-07-08 T 1 5 : 1 5 : 4 3 +0200</ d c t e r m s : c r e a t e d> <w l p : i s P r o d u c e d B y r d f : r e s o u r c e=" h t t p : // weblab . ow2

  o r g / o n t o l o g y#" /> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d f -schema# ">D e m o c r a t i c S o c i a l i s t R e p u b l i c o f S r i Lanka</ r d f s : l a b e l> <g e o : n a m e x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#"> D e m o c r a t i c S o c i a l i s t R e p u b l i c o f S r i Lanka</ g e o : n a m e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#k e r a l a ">

  l a b e l> <g e o : n a m e x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#">S t a t e o f KarnÄĄtaka</ g e o : n a m e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#s r i _ l a n k a "> <g e o : h a s S p a t i a l T h i n g r d f : r e s o u r c e=" h t t p : // sws . geonames . o r g / 1 2 2 7 6 0 3 / " x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // sws . geonames . o r g / 6 2 5 5 1 4 7 / "> < r d f : t y p e r d f : r e s o u r c e=" h t t p : //www. geonames . o r g / o n t o l o g y# F e a t u r e " /> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d f -schema# ">A s i a</ r d f s : l a b e l> <g e o : n a m e x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#">A s i a< / g e o : n a m e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // sws . geonames . o r g / 1 2 5 2 8 8 1 / "> < r d f : t y p e r d f : r e s o u r c e=" h t t p : //www. geonames . o r g / o n t o l o g y# F e a t u r e " />

  o r g / o n t o l o g y#" /> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d f -schema# ">R e p u b l i c o f I n d i a</ r d f s : l a b e l> <g e o : n a m e x m l n s : g e o=" h t t p : //www. geonames . o r g / o n t o l o g y#"> R e p u b l i c o f I n d i a</ g e o : n a m e> </ r d f : D e s c r i p t i o n> </ rdf:RDF> </ d a t a> </ a n n o t a t i o n> <m e d i a U n i t x s i : t y p e=" n s 3 : T e x t " u r i=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0 e b b 0 c 2 3 8 e 3 9#1"> APPENDIX E. FULL WEBLAB RESOURCE <a n n o t a t i o n u r i=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-a0 "> <d a t a x m l n s : w l=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / model#" x m l n s : r e s o u r c e C o n t a i n e r=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / s e r v i c e s / r e s o u r c e c o n t a i n e r "> <rdf:RDF x m l n s : r d f=" h t t p : //www. w3 . o r g /1999/02/22 -r d f -s y n t a x-n s#"> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1"> <d c : l a n g u a g e x m l n s : d c=" h t t p : // p u r l . o r g / dc / e l e m e n t s / 1 . 1 / "> en</ d c : l a n g u a g e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-a0 "> <d c t e r m s : c r e a t e d x m l n s : d c t e r m s=" h t t p : // p u r l . o r g / dc / t e r m s / ">2014-07-08 T 1 5 : 1 5 : 4 3 +0200</ d c t e r m s : c r e a t e d> <w l p : i s P r o d u c e d B y r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / s e r v i c e s#L a n g u a g e E x t r a c t i o n " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> </ rdf:RDF> </ d a t a> </ a n n o t a t i o n> <a n n o t a t i o n u r i=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-a1 "> <d a t a x m l n s : w l=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / model#" x m l n s : r e s o u r c e C o n t a i n e r=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / s e r v i c e s / r e s o u r c e c o n t a i n e r "> <rdf:RDF x m l n s : r d f=" h t t p : //www. w3 . o r g /1999/02/22 -r d f -s y n t a x-n s#"> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-2"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#m i d n a p o r e _ d i s t r i c t " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#p u r u l i a _ d i s t r i c t "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">P u r u l i a d i s t r i c t</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# P l a c e " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#k u l a b h e d a _ v i l l a g e "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">Kulabheda v i l l a g e</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# P l a c e " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#p l u s _ b o o k _ r e v i e w _ b u s i n e s s _ s c i t e c h _ n x g _ f r i d a y _ r e v i e w "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e#U n i t " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-57"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / TroopMovementOperation#d e p l o y e d " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / DeathEvent#de7948b6-f 4 b 7 -43c1 -910c -98 c a 2 3 f 9 0 f 7 3 "> < w o o k i e : i n v o l v e s r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#w e s t _ b e n g a l " x m l n s : w o o k i e=" h t t p : // weblab . ow2 . o r g / w o o k i e#" /> <w o o k i e : t a k e s P l a c e A t r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#w e s t _ b e n g a l " x m l n s : w o o k i e=" h t t p : // weblab . ow2 . o r g / w o o k i e#" /> 185 </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-43"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#b i h a r " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#w e s t _ b e n g a l "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">West B e n g a l</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# P l a c e " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#i n d i a "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">I n d i a</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# P l a c e " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-37"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#b i h a r " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / M e e t i n g#v i s i t s _ h i l l _ r e g i o n _ i n _ s r i _ l a n k a "> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#"> v i s i t s h i l lr e g i o n i n S r i Lanka</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# M e e t i n g " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / TroopMovementOperation#d e p l o y e d "> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">d e p l o y e d</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# TroopMovementOperation " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-15"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#b i h a r " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-23"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#m a g a z i n e _ l i t e r a r y _ r e v i e w _ m e t r o _ p l u s _ o p e n _ p a g e _ e d u c a t i o n " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-56"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / DamageEvent#damaged " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-42"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#b i h a r " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-22"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#p l u s _ b o o k _ r e v i e w _ b u s i n e s s _ s c i t e c h _ n x g _ f r i d a y _ r e v i e w " x m l n s : w l p =" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">B i h a r</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# P l a c e " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-35"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#p a k i s t a n " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#e r g a d a _ v i l l a g e "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">Ergada v i l l a g e</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# P l a c e " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / TroopMovementOperation#86 a c 1 e b e -6e f 3 -4a88-a877-d e 8 f e 4 8 9 e 2 c 7 "> < w o o k i e : i n v o l v e s r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#p o l i c e " x m l n s : w o o k i e=" h t t p : // weblab . ow2 . o r g / w o o k i e#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-17"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#w e s t _ b e n g a l " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-25"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#l a h o r e _ c o u r t " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#q u e s t "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">Quest</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e#U n i t " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-54"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#w e s t _ b e n g a l " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#k e r a l a "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">K e r a l a</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# P l a c e " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#jharkhand_party_member "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">Jharkhand P a r t y member</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e#U n i t " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-3"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#p u r u l i a _ d i s t r i c t " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-44"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P e r s o n#pankaj_kamboj " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / K i d n ap p i n gEve n t#cb6d501a -1cb7 -4223-ae30 -7 a 9 d 8 c 0 f 7 7 e f "> < w o o k i e : i n v o l v e s r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#e r g a d a _ v i l l a g e " x m l n s : w o o k i e=" h t t p : // weblab . ow2 . o r g / w o o k i e#" /> < w o o k i e : i n v o l v e s r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#jharkhand_party_member_kanai_murmu " x m l n s : w o o k i e=" h t t p : // weblab . ow2 . o r g / w o o k i e#" /> <w o o k i e : t a k e s P l a c e A t r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#e r g a d a _ v i l l a g e " x m l n s : w o o k i e=" h t t p : // weblab . ow2 . o r g / w o o k i e#" /> <w o o k i e : d a t e x m l n s : w o o k i e=" h t t p : // weblab . ow2 . o r g / w o o k i e#" >Monday</ w o o k i e : d a t e> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-34"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#d e l h i " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-24"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#mps_team" x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-16"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#w e s t _ b e n g a l " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#communist_party_of_india "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">Communist P a r t y o f I n d i a</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e#U n i t " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#m a g a z i n e _ l i t e r a r y _ r e v i e w _ m e t r o _ p l u s _ o p e n _ p a g e _ e d u c a t i o n "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e</ w l p : i s C a n d i d a t e> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">Magazine L i t e r a r y Review Metro P l u s Open Page E d u c a t i o n</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e#U n i t " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-26"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#union_government " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / DeathEvent#k i l l e d "> < r d f s : l a b e l x m l n s : r d f s=" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#"> k i l l e d</ r d f s : l a b e l> < r d f : t y p e r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e# DeathEvent " /> </ r d f : D e s c r i p t i o n> < r d f : D e s c r i p t i o n r d f : a b o u t=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-10"> <w l p : r e f e r s T o r d f : r e s o u r c e=" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#b u s i n e s s _ l i n e " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> </ r d f : D e s c r i p t i o n> p r o c e s s i n g #"/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#l a h o r e _ c o u r t "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e </ w l p : i s C a n d i d a t e > < r d f s : l a b e l x m l n s : r d f s =" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">L a h o r e c o u r t </ r d f s : l a b e l > < r d f : t y p e r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e#U n i t "/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P e r s o n#s i n g h _ l a y a "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e </ w l p : i s C a n d i d a t e > < r d f s : l a b e l x m l n s : r d f s =" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">S i n g h Laya</ r d f s : l a b e l > < r d f : t y p e r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e# P e r s o n "/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#b o k a r o _ d i s t r i c t "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e </ w l p : i s C a n d i d a t e > < r d f s : l a b e l x m l n s : r d f s =" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">Bokaro d i s t r i c t </ r d f s : l a b e l > < r d f : t y p e r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e# P l a c e "/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#p o l i c e "> <w l p : i s C a n d i d a t e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">t r u e </ w l p : i s C a n d i d a t e > < r d f s : l a b e l x m l n s : r d f s =" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">P o l i c e </ r d f s : l a b e l > < r d f : t y p e r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e#U n i t "/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-52"> <w l p : r e f e r s T o r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / P l a c e#s r i _ l a n k a " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g #"/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-12"> <w l p : r e f e r s T o r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#hindu_group " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g #"/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-28"> <w l p : r e f e r s T o r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / U n i t#i n d i a _ s _ n a t i o n a l _ n e w s p a p e r _ t u e s d a y " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g #"/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-59"> <w l p : r e f e r s T o r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / DeathEvent#k i l l e d " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g #"/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / K i d n ap p i n gEve n t#k i d n a p p e d"> < r d f s : l a b e l x m l n s : r d f s =" h t t p : //www. w3 . o r g / 2 0 0 0 / 0 1 / r d fschema#">kidnapped </ r d f s : l a b e l > < r d f : t y p e r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e# K i d n ap p in gEv e n t "/> </ r d f : D e s c r i p t i o n > < r d f : D e s c r i p t i o n r d f : a b o u t =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#1-60"> <w l p : r e f e r s T o r d f : r e s o u r c e =" h t t p : // weblab . ow2 . o r g / w o o k i e / i n s t a n c e s / DeathEvent#k i l l e d " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g #"/>

  t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-41"/> . I n f o r m e d s o u r c e s s a i d a n o t h e r member o f t h e p a r t y , Ananda Mahato , was a b d u c t e d by M a o i s t s from Kulabheda v i l l a g e i n t h e same Midnapore d i s t r i c t . Both Murmu and Mahato were members o f t h e a n t i -M a o i s t " Maobadi P r o t i r o d h Committee . " At Ghatbera i n P u r u l i a d i s t r i c t o f West Bengal , M a o i s t s r a n s a c k e d t h e h o u s e o f a l o c a l CPI ( M a r x i s t ) l e a d e r and s h o t and i n j u r e d a member o f t h e v i l l a g e r e s i s t a n c e c o m m i t t e e . As t h e a t t a c k e r s f a i l e d t o f i n d Chandan S i n g h Laya , CPI ( M a r x i s t ) Ghatbera l o c a l c o m m i t t e e s e c r e t a r y , t h e y s e t h i s h o u s e a b l a z e . I n Jharkhand , M a o i s t s blew up r a i l t r a c k s a t J h a r a n d i h i n Dhanbad , r e s u l t i n g i n t h e S h a k t i p u n j E x p r e s s and some l o c a l t r a i n s b e i n g h e l d up a t v a r i o u s p o i n t s , S e n i o r P u b l i c R e l a t i o n s O f f i c e r o f Dhanbad R a i l D i v i s i o n Amrendra Das s a i d a d d i n g a l i g h t e n g i n e d e r a i l e d a f t e r a one-and-h a l f m e t r e s e c t i o n o f t h e t r a c k was blown up . A g r o u p o f 12 M a o i s t s s e t t h r e e t r u c k s a b l a z e i n G i r i d i h d i s t r i c t ' s I s r i a r e a and b l o c k e d t h e Dumri-G i r i d i h r o a d w i t h f e l l e d t r e e s . M a o i s t s a l s o p a r t i a l l y b l a s t e d a r o a d b r i d g e c o n n e c t i n g Dumri t o t h e Grand Trunk Road and g u n s h o t s were heard , G i r i d i h S u p e r i n t e n d e n t o f P o l i c e Ravi Kant Dhan s a i d . H a z a r i b a g h S u p e r i n t e n d e n t o f P o l i c e Pankaj Kamboj s a i d M a o i s t s p a r t i a l l y damaged a r o a d w i t h e x p l o s i v e s a t S a r d a l o , b o r d e r i n g Bokaro d i s t r i c t . I n Bih ar , M a o i s t s b l a s t e d t h e t e l e c o m t o w e r a t S a l a i y a v i l l a g e . M a o i s t s a l s o dug up a 15-m e t r e s t r e t c h o f a r o a d a t Chanda v i l l a g e , d i s r u p t i n g t r a f f i c between Deo and Dhibra . They l e f t b e h i n d pamp hle ts , c l a i m i n g r e s p o n s i b i l i t y f o r t h e a c t i o n , o f f i c i a l s o u r c e s s a i d . B i h a r A d d i t i o n a l D i r e c t o r -G e n e r a l o f P o l i c e ( H e a d q u a r t e r s ) Neelmani s a i d t i g h t s e c u r i t y a r r a n g e m e n t s were i n p l a c e t o c o u n t e r t h e M a o i s t bandh . -PTI P r i n t e r f r i e n d l y page Send t h i s a r t i c l e t o F r i e n d s by E-M a

  t a x e s t o f u n d f l o o d r e l i e f C a n d i d a t e s D i r e c t i v e t o f o l l o w p a n e l g u i d e l i n e s P a k i s t a n i d i p l o m a t e l e c t r o c u t e d Kidnapped D e l h i c h i l d f o u n d dead i n r e l a t i v e ' s c a r P e t i t i o n f i l e d a g a i n s t N i t i s h P o l i c e d e p l o y e d i n K a i t h a l D e l h i can d e l i v e r g r e a t Games: F e n n e l l P a k i s t a n d i p l o m a t d i e s o f e l e c t r i c s h o c k Worker d i e s a f t e r i n h a l i n g p o i s o n o u s g a s e s 10 k i l l e d i n UP f l o o d s F e n n e l l c o n f i d e n t C a s e s withdrawn One-way s o o n a t L a k d i k a p u l A r c h i v e s Y e s t e r d a y ' s I s s u e D a t e w i s e F e a t u r e s : Magazine L i t e r a r y Review Metro P l u s Open Page E d u c a t i o n P l u y r i g h t Âľ 2 0 0 9 , The Hindu . R e p u b l i c a t i o n o r r e d i s s e m i n a t i o n o f t h e c o n t e n t s o f t h i s s c r e e n a r e e x p r e s s l y p r o h i b i t e d w i t h o u t t h e w r i t t e n c o n s e n t o f The Hindu</c o n t e n t > </mediaUnit> </ r e s o u r c e > Listing E.1 -An example of a full WebLab resource.

F

FFigure

  Figure F.1 -Une chaîne WebLab minimale : un document d est converti en une ressource XML r 0 et les ensembles d'annotations u i sont ajoutés par chaque service.

Algorithm 10 :

 10 Q-learning Data: MDP M 1 while vrai do 2 Choisir a t en utilisation la stratégie d'exploration EG 3 Exécuter a t , observer s t+1 4

  Notre approche fonctionne très bien avec du feedback numérique (section F.7), mais très peu d'utilisateurs pourraient dire « l'extraction d'un événement incorrect vaut 10,5 fois mieux qu'une extraction manquée, qui elle-même vaut 7,9 fois mieux qu'une extraction prenant 90 secondes ». Plus vraisemblablement, les utilisateurs peuvent affirmer « je préfère l'extraction d'un événement incorrect à une extraction manquée (car je ne veux pas perdre d'information), et je préférerais que l'extraction soit aussi rapide que possible ». Notons ici l'expression de deux préférences, qui, bien qu'elles ne soient pas indépendantes (on peut supposer qu'une extraction plus rapide soit de moins bonne qualité), ne peuvent pas facilement être reliées numériquement. Nous voulons donc permettre aux utilisateurs de donner des retours à l'agent de façon indépendante sur les deux dimensions : « je préfère qu'un événement soit extrait », et « je préfère que l'extraction soit aussi rapide que possible ». De telles préférences peuvent être représentées par un ordre partiel sur les résultats possibles de l'extraction tels que perçus par l'utilisateur, notés f 1 , . . . , f k et vus comme des états finaux du MDP. Par exemple, f i peut être « extraction correcte en 10 secondes », ou encore « traitement stoppé après 5 secondes, sans événement extrait ».

F

  .6. MESURE DE LA QUALITÉ DES RÉSULTATS 217 F.6 Mesure de la qualité des résultats Pour mesurer la qualité des résultats de nos tests, il faut d'abord définir la similarité entre un événement extrait par la chaîne sur le documentE 1 = (C 1 , T 1 , G 1 , A 1 ) et l'événement « parfait » du GTD (l'oracle) E 2 = (C 2 , T 2 , G 2 , A 2 ) qui correspond à ce document.De nombreuses méthodes ont été proposées pour mesurer la similarité[START_REF] Pandit | A comparative study on distance measuring approaches for clustering[END_REF];[START_REF] Tversky | Studies of similarity[END_REF];[START_REF] Cohen | Very Fast Similarity Queries on Semi-Structured Data from the Web[END_REF];[START_REF] Dutkiewicz | Knowledge-based highly-specialized terrorist event extraction[END_REF], pour n'en citer que quelques-unes), mais nous avons besoin d'une mesure qui prenne en compte les quatre dimensions des événements, et sensible par ailleurs à la différence entre différents types d'événements. Nous définissons donc :σ(E 1 , E 2 ) = aσ C (C 1 , C 2 ) + bσ T (T 1 , T 2 ) + cσ G (G 1 , G 2 ) + dσ A (A 1 , A 2 ) (a + b + c + d) La similarité conceptuelle σ C (C 1 , C 2 ) est de 1 s'il y a un atome commun entre les dimensions conceptuelles de E 1 et E 2 , et de 0 sinon. Par exemple, pour C 1 = {BombingEvent, AttackEvent} et C 2 = {BombingEvent}, on obtient C 1 ∩ C 2 = {BombingEvent}, et donc σ C (C 1 , C 2 ) = 1.Nous procédons de même pour la similarité géographique σ G (G 1 , G 2 ).

  Example 16 (Similarité) Prenons les deux événements qui sont décrits dans la Figure F.3. Nous voyons un atome commun ( Death Event) dans la dimension conceptuelle, qui donne une similarité de 1. La similarité spatiale est de 1 grâce à l'élément West Bengal. Nous déduisons le jour de la semaine, pour obtenir une similarité temporelle de 0.143. Enfin, la similarité agentive, comme nous avons déjà vu, est de 0.5. Avec des poids égaux sur les dimensions (a = b = c = d = 1), nous avons une similarité globale entre l'événement extrait, et celui du GTD de σ(WebLab Event, GTD Event)

F. 7 . 1 Figure

 71 Figure F.4 -Pourcentage d'événements extraits par l'IA non-formée.

F

  .7.2 Tests de performance sur des documents avec un très faible pourcentage d'événements extractibles.Pour ces tests, nous avons augmenté l'espace des actions en donnant à l'IA le choix parmi dix gazetteers (cinq listes de verbes et cinq de noms) : six (les verbes et noms de bombing, shooting et injure) contiennent des mots susceptibles de déclencher la détection d'un événement de ce type, deux (les verbes et noms de mixte) contiennent un mélange de mots susceptibles de déclencher la détection de plusieurs types d'événements, et deux sont factices. Nous nous attendions à ce que l'IA apprenne à utiliser de préférence les gazetteers mixte, ensuite les gazetteers qui déclenchent la détection d'un événement unique, et qu'elle évite les gazetteers factices. Nous nous attendions également à ce que l'IA ne soit pas sensible à un espace d'actions plus grand.Nous avons testé une IA non-formée cette fois-ci sur 5 000 documents inconnus, aléatoirement choisis avec en moyenne 22.56% d'événements extractibles(Figure F.5).

FigureFigureFFigure

  Figure F.7 -La qualité des extractions de la chaîne « experte », comparées à celles de l'IA non-formée pour les documents 1 -2 400.

F

  .7.3 Tests de performance avec un feedback sporadique Les deux jeux de tests précédents avaient pour but d'évaluer la capacité de l'IA à extraire des événements depuis des documents inconnus. Nous avons réalisé une troisième série de tests afin d'évaluer la vitesse à laquelle une IA non-formée pourrait apprendre la politique d'une chaîne experte (capable d'extraire 100 % des événements), et la qualité de cette politique, pour vérifier que les événements extraits sont bien similaires aux résultats attendus. Pour cela, nous avons pris un jeu de 63 documents de la GTD, chacun d'eux contenant un événement extractible. Nous avons traité ces documents dans le même ordre, de manière répétée, par lots. Nous avons utilisé le même espace d'actions qu'en sous-section F.7.2, c'est-à-dire, un choix entre les services, STOP et dix gazetteers (cinq listes de verbes, et cinq listes de noms). Le Q-learning a été lancé avec des paramètres similaires à ceux des premiers tests : ǫ = 0.4, divisé par 2 tous les 10 lots jusqu'à ǫ = 0.1, α = 0.2, et λ = 0.Nous avons d'abord entraîné l'IA avec une récompense donnée pour chaque document. Il s'est avéré qu'après seulement 1 008 documents (c'està-dire 16 lots), la politique apprise était capable d'extraire 100 % des événements. De manière plus importante, cette politique était optimale, c'està-dire que la qualité de l'extraction était à égalité avec celle de la chaîne experte, comme l'illustre la Figure F.10a. L'analyste n'est pas disponible en permanence pour consulter chaque document dès qu'il est traité. Nous avons alors lancé un test similaire, mais la récompense n'était donnée qu'à la fin du traitement de chaque lot, empêchant l'IA d'apprendre durant un lot. L'IA était légèrement plus lente à apprendre une politique optimale, ayant besoin de 1 260 documents (20 lots, Figure F.10b).

FFigure

  Figure F.10 -La qualité de la politique apprise avec une récompense (a) complète instantanée ; (b) complète différée ; (c) sporadique à 50% extraction ; (d) sporadique à 100% extraction.

  Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Introducing Dora . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Formalisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.1 Q-Learning(λ) . . . . . . . . . . . . . . . . . . . . . . 12.3.2 Dora . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.1 Random MDP tests . . . . . . . . . . . . . . . . . . . 12.4.2 Cliff-walk MDP tests . . . . . . . . . . . . . . . . . . . 12.4.3 Naive QL(λ) . . . . . . . . . . . . . . . . . . . . . . . 12.5 Experience replay . . . . . . . . . . . . . . . . . . . . . . . . . 12.6 Related and future work . . . . . . . . . . . . . . . . . . . . . 12.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . 14.1.2 Implications . . . . . . . . . . . . . . . . . . . . . . . . 14.1.3 Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . Percentage of documents containing extractable events by 100 documents for tests with few extractable events. . . . . . . . . 78 8.3 Percentage of events extracted using the policy learnt after each 100 documents. . . . . . . . . . . . . . . . . . . . . . . . 79 8.4 Extraction quality for the expert chain against that of the untrained AI for documents 1 -2 400 with a very low percentage of extractable events. . . . . . . . . . . . . . . . . . . . . . . . 80 8.5 Extraction quality for the expert chain against that of the untrained AI for documents 2 401 -4 800 with a very low percentage of extractable events. . . . . . . . . . . . . . . . . . . 81 8.6 Extraction quality for the expert chain against that of the untrained AI for documents 4 800 -5 000 with a very low percentage of extractable events. . . . . . . . . . . . . . . . . . . 82 8.7 Odd behaviour from the AI with full feedback and ǫ divided by 2 every 100 documents to a minimum value of 0.1. . . . . . 84 8.8 The state s j is pivotal. Explore actions from it can reduce its perceived value until the best action in s i is STOP. . . . . . . 85 8.9 Even worse behaviour from the AI with full feedback and increased exploration as ǫ is divided by 2 every 200 documents. 85 8.10 Full feedback with aggressive non-exploration strategies based on the number of visits to each state. . . . . . . . . . . . . . . 86 8.11 Full feedback with an exploration strategy based on the number of visits to each state where β = 0.55. . . . . . . . . . . . 86 8.12 Full feedback delayed until 100 documents have been treated. 87 8.13 Partial feedback delayed until 100 documents have been treated. 88 9.1 Roll-outs are carried out from s 1 , giving a preference order: τ 1 , τ 2 , τ 3 over the trajectories, and hence over the actions from s 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 9.2 Two policies, where we have defined the final states in terms of wealth levels, or rewards. . . . . . . . . . . . . . . . . . . . 94 9.3 A potential partial preference relation over final state criteria. 96 9.4 Numerical rewards received by the AI after training over 30 ×100 documents with immediate feedback, compared with the expert chain. . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 9.5 The player first chooses not to throw die A, then chooses to throw die B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

	vi CONTENTS II Further contributions xii 8.2 LIST OF FIGURES	FOREWORD Contents ix LIST OF FIGURES
	12 Dora -exploiting QL explorations
	open-source. Abstract 12.1 Appendices	other services that I used are i
	A An example configuration
	B Wookie Ontology
	v

13 Application to image analysis 13.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2 Title detection . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3 Application of BIMBO . . . . . . . . . . . . . . . . . . . . . . 13.4 Choice of states and actions . . . . . . . . . . . . . . . . . . . 13.5 Similarity measure . . . . . . . . . . . . . . . . . . . . . . . . 13.6 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III Conclusion 14 Discussion and perspectives 14.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.1.1

  ).

	CHAPTER 5. REINFORCEMENT LEARNING
	a0: milk?
	s0:
	empty
	basket
	a1: grapes?

state s 269 , it chooses to change the nouns gazetteer again, this time for the list of bombing nouns, taking it to state s 194 .

	7.2. AN EXAMPLE DECISION PROCESS FOR ONE DOCUMENT 55
	s 0	MixedVerbs	s 84	Tika	s 268
			Geo		DummyNouns
	s 149	BombingVerbs	s 194	BombingNouns	s 269
	NgramJ
	s 47	InjureVerbs	s 187	MixedVerbs	s 180
					ShootingNouns
	stop	STOP	s 379	Gate	s 179
	Figure 7.2 -The graphical representation of the choices made during the
	treatment of a single document roughly halfway through the learning process.
	Similarly, from

Table 7 .

 7 1 -An abridged example of the fields from several GTD events. Kanai Murmu (48), was found. Murmu, a resident of Jorkisoli village near Lalgarh, was dragged out from his house by a group of armed men in the early hours of 10/12/2009. Later, his bullet-riddled body was found by local residents beside a pond in Binapur's Salpatra.

	value

  The state s j is pivotal. Explore actions from it can reduce its perceived value until the best action in s i is STOP.

	s 1	s i	a i	s j	treatment	s n	extraction
		STOP
				s k
	Figure 8.8 -
							, Chapter 2) noted a similar

  From state s 1 , the agent exploits action a 0 , then from state s2 it exploits action a 1 , and finally, from state s 3 , it exploits action a 2 .

						a 3 : 0	s 5	1000
	s 1	a 0 : 0	s 2	a 1 : 0	s 3	a 2 : 0	s 4	10
		(a) The world -state s 5 rewards much more than state s 4
						a 3 : 0	s 5	1000
	s 1	a 0 : 0	s 2	a 1 : 0	s 3	a 2 : 0	s 4	10
	(b) s 1	a 0 : 10	s 2	a 1 : 10	s 3	a 2 : 10	s 5 s 4
	• Figure 12.4e: It receives the jackpot of 1000, which cannot now be
	back-propagated. The action a 0 in state s 1 is still apparently worth 10.

  From state s 1 , Dora exploits action a 0 , then from state s2 she exploits action a 1 , and finally, from state s 3 , she exploits action a 2 .

						a 3 : 0	s 5	1000
	1	a 0 : 0	s 2	a 1 : 0	s 3	a 2 : 0	s 4	10
		(a) The world -state s 5 rewards much more than state s 4
						a 3 : 0	s 5	1000
	s 1	a 0 : 0	s 2	a 1 : 0	s 3	a 2 : 0	s 4	10
	(b) s 1	a 0 : 10	s 2	a 1 : 10	s 3	a 2 : 10	s 5 s 4

  was an exploit, add to trace, back-propagate temporal difference

		128	CHAPTER 12. DORA -EXPLOITING QL EXPLORATIONS
		Algorithm 6: Function QL-Back-propagate(trace, δ t )
		Data: trace, δ t
		1
	5 else
	6	//a was an explore, clear trace, update one time-step,
		add to trace
	7	trace.clear()
	8	Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )δ t
	9	trace.add(s t , a t )

3

trace.add(s t , a t ) 4 QL-Back-propagate(trace, δ t ) using Algorithm 6

  Given: s t , Qt (s t , a t ) ; Played: a t ; Observed:s t+1 , r t+1 1 δ t ← r t+1 + γmax b {Q t (s t+1 , b)} -Qt (s t , a t ) , a t ) + α t (s t , a t )δ t max Qt (s t , a t ) + α t (s t , a t )δ tmax

	130		CHAPTER 12. DORA -EXPLOITING QL EXPLORATIONS
	Algorithm 8: Dora: update Q-Values
	Data: 2 if a t was argmax b	{ Qt (s t , b)} then
		//a was an exploit, treat as in QL(lambda)
	3	trace.add(s t , a t )
	4	Dora-Back-propagate(trace, δ t , 0) using Algorithm 9
	5 else
	6		Qt+1
	7	if	Qt (s t b	{ Qt (s t , b)} then
			//a is now the new best action in s_t
					b)}
	and back-propagate it down the trace.

8 ∆ t ← b { Qt (s t , b)} 9 Dora-Back-propagate(trace, ∆ t , 1) using Algorithm 9 10 Qt+1 (s, a) ← Qt (s t , a t ) + α t (s t , a t )δ t 11 trace.add(s t , a t ) 12 else //a was an explore and is still worse than the best action, treat as in QL(lambda) 13 trace.clear() 14 Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )δ t 15 trace.add(s t , a t ) Algorithm 9: Function Dora-Back-propagate(trace, δ, exponent 0 ) Data: trace, δ t , exponent 0 1

  . Cet algorithme traite le cas général d'un critère SSB. Comme le Q-learning, il utilise l'exploration EG, et il met à jour les Q-valeurs d'une manière similaire. Toutefois, au lieu d'utiliser du feedback numérique, que l'utilisateur n'a pas à lui fournir, il utilise φ et ses traitements passés, précisément les fréquences p t avec lesquelles il a obtenu chaque résultat possible jusqu'alors (la valeur numérique ainsi définie est notée R pt (s t+1 )). Intuitivement, l'algorithme cherche continuellement à faire mieux, au sens de φ, qu'il n'a fait jusqu'alors. Le pseudo-code de SSB Q-learning est donné par l'algorithme 11.Qt+1 (s t , a t ) ← Qt (s t , a t ) + α t (s t , a t )(r t+1 + max t+1 est un état final f i et que l'agent n'a pas exploré then

	Algorithm 11: SSB Q-learning
		Data: MDP M, fonction SSB φ
	1 while vrai do
	2	Choisir a t en utilisant la stratégie d'exploration EG
	3	Exécuter a t , observer s t+1
	4	Soit r t+1 = R pt (s t+1 )
	7	p t+1 = p t + 1 η+1 (1 i -p t )
	8	# η est le nombre de fois où p a été mis à jour
		Enfin, pour les deux algorithmes, nous avons utilisé des versions avec
	traces d'éligibilité

5 b { Qt (s t+1 , b)} -Qt (s t , a t )) 6 if s

  F.4. AMÉLIORATION CONTINUE VIA L'APPRENTISSAGE PAR RENFORCEMENT213 F.4 Amélioration continue via l'apprentissage par renforcement Puisque une chaîne unique pour traiter parfaitement chaque type de document est impossible à construire, l'idéal serait une chaîne faite sur mesure pour chaque document. Notre objectif est donc l'amélioration continue de la chaîne de traitement, de sorte que le système apprenne de ses erreurs. Une partie de notre contribution est de modéliser le traitement d'un document comme un processus décisionnel markovien (MDP), et son amélioration comme un problème d'apprentissage par renforcement. Une telle formalisation n'est pas directe, car le processus de traitement et ses entrées sont hétérogènes, et malgré tout ils doivent être formalisés en états et actions d'un problème de décision. De plus, les récompenses ne peuvent pas être spécifiées aisément, en particulier parce qu'elles doivent être récoltées de façon invisible pour les utilisateurs, sans impact sur leur travail. Finalement, la distribution des documents et les préférences des utilisateurs sont inconnues, et peuvent changer à tout moment.

  F.7. TESTS AVEC UN FEEDBACK NUMÉRIQUE Figure F.3 -Comparaison de deux événements.de la définition de la similarité et de ses paramètres, une tâche cognitive difficile. Notons que l'information donnée à l'IA, ici, est la même que celle utilisée pour l'évaluation de la qualité de la politique qu'elle a apprise. Cette méthode est classique en RL, mais pas naturelle dans la vie réelle. Dans la section F.8, nous retirons cette contrainte et produisons les résultats obtenus avec du feedback qualitatif.Nous avons constitué trois groupes de tests avec une récompense numérique :1. Une IA a été formée et sa performance a ensuite été comparée avec celle d'une IA non-formée, et d'une chaîne « experte » sur 1 000 documents inconnus, biaisés vers un type d'événement « intéressant ».2. La performance d'une IA non-formée a été testée sur 5 000 documents inconnus, choisis aléatoirement, avec un très faible pourcentage d'événements extractibles.3. Finalement, la capacité d'apprentissage d'une IA non-formée a été examinée avec une récompense sporadique.

						219
					KidnappingEvent
		killed			Hostage Taking (Kidnapping)
		MilitaryEvent			DeathEvent	India
	Jharkhand Party member	WebLab Event DeathEvent	West Bengal	Communist Party India-Maoist (CPI-Maoist)	GTD Event 10/12/2009	Jorkisoli village Lalgarh India Midnapore
		Monday		Jharkhand anti-Maoist Maobadi Protirodh Com-		South Asia
				mittee member
						West Bengal
				(b) L'événement extrait « parfait » du
	(a) L'événement extrait	GTD

  résumé, nous nous attendions à ce que QL soit plus efficace que MAG, et MAG plus efficace que DOM, étant donné la quantité d'information qu'ils reçoivent. Nous constatons, cependant, que QL peut donner d'excellents résultats, mais est sensible à la variation de paramètres et dépend du feedback numérique. DOM ne nécessite qu'un feedback purement ordinal, et pourtant, avec les bons paramètres, est capable d'apprendre des politiques optimales, montrant la faisabilité de cette approche. MAG se révèle la meilleure approche dans un contexte industriel, combinant les avantages des deux méthodes : robustesse au choix des paramètres, utilisation du feedback essentiellement intuitif, et capacité à apprendre très rapidement une politique optimale.

Of course, BIMBO being completely modular, slotting in a new similarity measure would have no impact on the rest of the framework.

Hugo Gilbert provided the technical proof, we provided the motivation and the experiments, and we all collaborated on the early design of the algorithm

Bien sûr, une telle information pourrait être utilisée afin de généraliser sur les actions par exemple, ou de construire un modèle a priori des actions et des états, mais nous laissons ceci pour un travail futur.

Ainsi, bien que les courbes rouges représentent toujours la même approche appliquée sur les mêmes documents, elles n'ont pas toujours la même allure puisque seuls des extraits en sont présentés.

Acknowledgements

Acknowledgements iii Foreword

CHAPTER 11. CONCLUSION TO PART I

Part II

Further contributions 119 Appendices APPENDIX A. AN EXAMPLE CONFIGURATION <p r o p e r t y name=" w r i t e S i m R e s u l t s " v a l u e="FALSE" /> < !--Whether t h e b o d i e s s h o u l d be s t o r e d o r n o t --> <p r o p e r t y name=" w r i t e B o d y " v a l u e="FALSE" /> < !--Whether t h e r e w a r d s s h o u l d be d e l a y e d u n t i l t h e end o f b a t c h (TRUE) o r g i v e n e v e r y document (FALSE) --> <p r o p e r t y name=" d e l a y R e w a r d s " v a l u e="FALSE" /> < !--I f t h e r e w a r d s a r e d e l a y e d u n t i l t h e end o f batch , what p e r c e n t a g e o f t h e r e w a r d s s h o u l d be g i v e n --> <p r o p e r t y name=" p e r c e n t a g e T o G i v e " v a l u e=" 100 " /> < !--Whether BIMBO s h o u l d s e n d E s t h e r p r o g r e s s r e p o r t s by SMS --> <p r o p e r t y name="sendSMS" v a l u e="FALSE" /> < !--I f so , e v e r y how many b a t c h e s ? --> <p r o p e r t y name=" sendSMSBatches " v a l u e=" 10 " /> </ bean> <bean i d=" C o n f i g P a t h s " c l a s s=" c o n f i g . C o n f i g P a t h s "> < d e s c r i p t i o n> C o n f i g u r a t i o n p a r a m e t e r s f o r t h e p a t h s and f i l e n a m e s n e e d e d by t h e AI . I f t h e b u n d l e t e s t i n g f o l d e r h a s been c o p i e d c o r r e c t l y , t h e s e s h o u l d n e v e r c h a n g e </ d e s c r i p t i o n> < !--SPARQL RESOURCES --> <p r o p e r t y name=" s p a r q l Q u e r y E n d p o i n t " v a l u e=" ${ s p a r q l E n d p o i n t }/KDB -i n f e r / q u e r y " /> <p r o p e r t y name=" s p a r q l U p d a t e E n d p o i n t " v a l u e=" ${ s p a r q l E n d p o i n t }/KDB/ u p d a t e " /> <p r o p e r t y name=" s p a r q l R e q u e s t s P a t h " v a l u e=" ${ b u n d l e T e s t i n g }/ r e q u e s t s / " /> < !--BUNDLE RESOURCES --> <p r o p e r t y name=" b u n d l e T e s t i n g " v a l u e=" ${ b u n d l e T e s t i n g } " /> <p r o p e r t y name=" b u n d l e F o l d e r " v a l u e=" ${ b u n d l e T e s t i n g }${ b u n d l e I n s t a n c e }/ " /> <p r o p e r t y name=" g a t e C o n f " v a l u e=" ${ b u n d l e T e s t i n g }${ b u n d l e I n s t a n c e }/ c o n f / s e r v i c e s / wookie-g a t ee x t r a c t i o n . c x f -s e r v l e t . xml " /> <p r o p e r t y name=" g a z e t t e e r P a t h " v a l u e=" ${ b u n d l e T e s t i n g }${ b u n d l e I n s t a n c e }/ c o n f / g a t e / p l u g i n s /EADS/ENG/ G a z e t t e e r s / E v e n t s / O s i n t E v e n t s / " /> < !--AI RESOURCES --> < !--L o c a t i o n o f t o I n d e x f o l d e r i n BIMBOs t r e a t m e n t c h a i n --> <p r o p e r t y name=" a i C h a i n T o I n d e x " v a l u e=" ${ b u n d l e T e s t i n g } a i -c h a i n -to-p r o c e s s / " /> < !--L o c a t i o n o f t h e c o p i e d GATE m a s t e r l i s t s m a n i p u l a t e d by t h e AI --> <p r o p e r t y name=" g a z e t t e e r L i s t P a t h " v a l u e=" ${ t h e s i s -a i R e s o u r c e s } g a z e t t e e r s / " /> < !--L o c a t i o n o f t h e r e s o u r c e s u s e d by t h e AI --> <p r o p e r t y name=" a i R e s o u r c e s P a t h " v a l u e=" ${ t h e s i s -a i R e s o u r c e s } " /> < !--Where t h e Camel R o u t e s a r e --> <p r o p e r t y name=" r o u t e s F i l e " v a l u e=" f i l e : ${ t h e s i s -a i R e s o u r c e s } DynamicRoutes . xml " /> < !--Where t h e GTD c s v f i l e s and s o u r c e p a g e s a r e s t o r e d --> <p r o p e r t y name=" g t d C s v L o c a t i o n s " v a l u e=" ${ b u n d l e T e s t i n g } p a g e s _ a l l _ e x t r a c t a b l e s " /> < !--TEST RESULTS --> < !--Where t h e r e s u l t s a r e s t o r e dt h e l e a r n t v a l u e s , r e w a r d s , c o n s o l e summary and e x p l o r e / e x p l o i t data , p l u s t h e e v e n t s e x t r a c t e d --> <p r o p e r t y name=" r e s u l t F i l e s " v a l u e=" ${ b u n d l e T e s t i n g } R e s u l t s / " /> < !--Where t h e b o d i e s a r e s t o r e dt h e WebLab r e s o u r c e --> <p r o p e r t y name=" b o d y F i l e s " v a l u e=" ${ b u n d l e T e s t i n g } Morgue / " /> < !--The b a s e name o f t h e a c t i o n / s t a t e / r e w a r d c s v f i l e w r i t t e n a t e a c h i t e r a t i o n , and a t t h e end t h e f i l e n a m e o f t h e l e a r n t v a l u e s --> <p r o p e r t y name=" l e a r n t V a l u e s F i l e n a m e " v a l u e=" l e a r n t V a l u e s " /> < !--Used t o d i s t i n g u i s h t h e f i n a l a c t i o n / s t a t e / r e w a r d c s v f i l e which i s u s e d t o r e s t a r t t h e run , from t h o s e w r i t t e n a t e a c h i t e r a t i o n --> <p r o p e r t y name=" r e s t a r t S t r i n g " v a l u e=" R e s t a r t " /> < !--The f i l e n a m e o f t h e QL a l g o r i t h m , s t o r e d f o r a r e p e a t run --> <p r o p e r t y name=" Q L S a v e F i l e " v a l u e="QL . s e r " /> < !--The f i l e n a m e o f t h e rMax a l g o r i t h m , s t o r e d f o r a r e p e a t run --> <p r o p e r t y name=" r M a x S a v e F i l e " v a l u e=" rMaxSave . s e r " /> < !--The f i l e n a m e o f t h e vMax a l g o r i t h m , s t o r e d f o r a r e p e a t run --> <p r o p e r t y name=" vM axS ave F il e " v a l u e=" vMaxSave . s e r " /> < !--The f i l e n a m e o f t h e SSBQL a l g o r i t h m , s t o r e d f o r a r e p e a t run --> <p r o p e r t y name=" SSBQLSaveFile " v a l u e="SSBQLSave .

Appendix D

Full web page

This web page is reproduced here with the kind permission of Kasturi & Sons Ltd (KSL). <w l p : h a s N a t i v e C o n t e n t r d f : r e s o u r c e=" f i l e : d a t a / c o n t e n t / weblab . 2 3 9 0 5 7 2 0 9 6 4 8 9 9 1 0 2 3 8 . c o n t e n t " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#" /> < w l p : h a s O r i g i n a l F i l e S i z e r d f : d a t a t y p e=" h t t p : //www. w3 . o r g / 2 0 0 1 /XMLSchema#l o n g " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">2 3 0 2 3</ w l p : h a s O r i g i n a l F i l e S i z e> <w l p : h a s O r i g i n a l F i l e N a m e x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">2 0 0 9 1 0 1 2 0 0 2 4_1 . html</ w l p : h a s O r i g i n a l F i l e N a m e> <d c : s o u r c e x m l n s : d c=" h t t p : // p u r l . o r g / dc / e l e m e n t s / 1 . 1 / ">d a t a / t o I n d e x / 2 0 0 9 1 0 1 2 0 0 2 4_1 . html</ d c : s o u r c e> <d c t : m o d i f i e d r d f : d a t a t y p e=" h t t p : //www. w3 . o r g / 2 0 0 1 /XMLSchema# dateTime " x m l n s : d c t=" h t t p : // p u r l . o r g / dc / t e r m s / ">2014-07-08 T 1 5 : 0 1 : 2 2 +02 : 0 0</ d c t : m o d i f i e d> <w l p : h a s G a t h e r i n g D a t e r d f : d a t a t y p e=" h t t p : //www. w3 . o r g / 2 0 0 1 / XMLSchema#dateTime " x m l n s : w l p=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / o n t o l o g y / p r o c e s s i n g#">2014-07-08 T 1 5 : 1 4 : 5 4 .719+02 : 0 0</ w l p : h a s G a t h e r i n g D a t e> </ r d f : D e s c r i p t i o n> </ rdf:RDF> </ d a t a> </ a n n o t a t i o n> <a n n o t a t i o n u r i=" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9#a2 "> <d a t a x m l n s : w l=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / model#" x m l n s : r e s o u r c e C o n t a i n e r=" h t t p : // weblab . ow2 . o r g / c o r e / 1 . 2 / s e r v i c e s / r e s o u r c e c o n t a i n e r "> <rdf:RDF x m l n s : r d f=" h t t p : //www. w3 . o r g /1999/02/22 -r d f -s y n t a x -n s#"> < d c t : s p a t i a l x m l n s : d c t =" h t t p : // p u r l . o r g / dc / t e r m s / " > 2 0 . 0 , 7 7 . 0 < / d c t : s p a t i a l > < d c t : s p a t i a l x m l n s : d c t =" h t t p : // p u r l . o r g / dc / t e r m s / " > 1 0 . 0 , 7 6 . 5 < / d c t : s p a t i a l > < d c t : s p a t i a l x m l n s : d c t =" h t t p : // p u r l . o r g / dc / t e r m s / " > 7 . 0 , 8 1 . 0 < / d c t : s p a t i a l > < d c t : s p a t i a l x m l n s : d c t =" h t t p : // p u r l . o r g / dc / t e r m s / " > 2 4 . 0 , 8 8 . 0 < / d c t : s p a t i a l > < d c t : s p a t i a l x m l n s : d c t =" h t t p : // p u r l . o r g / dc / t e r m s / " > 1 3 . 5 , 7 6 . 0 < / d c t : s p a t i a l > < d c t : s p a t i a l x m l n s : d c t =" h t t p : // p u r l . o r g / dc / t e r m s / " > 2 8 . 6 6 6 7 , 7 7 . 1 < / d c t : s p a t i a l > </ r d f : D e s c r i p t i o n > </rdf:RDF> </data> </ a n n o t a t i o n > <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="3333" end ="3342" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-30"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="3634" end ="3645" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-54"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="3658" end ="3663" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-29"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="3927" end ="3939" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-25"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="3975" end ="3983" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-24"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="3984" end ="4015" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-55"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4006" end ="4015" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-52"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4057" end ="4062" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-32"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4092" end ="4100" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-31"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4456" end ="4461" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-34"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4533" end ="4539" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-48"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4540" end ="4548" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-57"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4563" end ="4568" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-33"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4605" end ="4613" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-35"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4851" end ="4910" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-23"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="4911" end ="4967" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-22"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="5010" end ="5015" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-49"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="5052" end ="5063" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-12"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="5124" end ="5151" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-13"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="5188" end ="5201" u r i =" h t t p : // r e s o u r c e _ a e c a 4 2 5 2 -74 f d -427 f -a697 -0e b b 0 c 2 3 8 e 3 9 #1-10"/> <segment x s i : t y p e =" n s 3 : L i n e a r S e g m e n t " s t a r t ="5205" end ="5225" u r i =" h t

Appendix E

Full WebLab resource