Jaakko Hollmén
email: jaakko.hollmen@aalto.fi

Panagiotis Papapetrou
email: panagiotis@dsv.su.se

Clustering diagnostic profiles of patients

Keywords: medical records, binary representations, clustering

Electronic Health Records provide a wealth of information about the care of patients and can be used for checking the conformity of planned care, computing statistics of disease prevalence, or predicting diagnoses based on observed symptoms, for instance. In this paper, we explore and analyze the recorded diagnoses of patients in a hospital database in retrospect, in order to derive profiles of diagnoses in the patient database. We develop a data representation compatible with a clustering approach and present our clustering approach to perform the exploration. We use a k-means clustering model for identifying groups in our binary vector representation of diagnoses and present appropriate model selection techniques to select the number of clusters. Furthermore, we discuss possibilities for interpretation in terms of diagnosis probabilities, in the light of external variables and with the common diagnoses occurring together.

Introduction

Electronic Health Records (EHR) provide a wealth of information for retrospective analysis of patients. They can be a source for validating the conformance to planned treatment, or can be used to mimic the doctor by predicting diagnoses of patients with the use of vital signs and other symptoms. The main adoption of EHRs in healthcare research has been rapidly increasing [START_REF] Hersh | Adding value to the electronic health record through secondary use of data for quality assurance, research, and surveillance[END_REF][START_REF] Weiskopf | Defining and measuring completeness of electronic health records for secondary use[END_REF]. In contrast to traditional data sources, including spontaneous reports [START_REF] Van Puijenbroek | A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions[END_REF] or social media [START_REF] Sarker | Utilizing social media data for pharmacovigilance: a review[END_REF], EHRs comprise disparate data types and can convey critical information which could potentially allow medical practitioners to prevent critical conditions or provide a timely intervention when necessary. A wide body of research using supervised learning approaches on EHR data exists in the literature, e.g., [START_REF] Harpaz | Mining electronic health records for adverse drug effects using regression based methods[END_REF][START_REF] Henriksson | Identifying adverse drug event information in clinical notes with distributional semantic representations of context[END_REF][START_REF] Park | A novel algorithm for detection of adverse drug reaction signals using a hospital electronic medical record database[END_REF][START_REF] Schuemie | Using electronic health care records for drug safety signal detection: a comparative evaluation of statistical methods[END_REF][START_REF] Zhao | Predictive modeling of structured electronic health records for adverse drug event detection[END_REF], mostly focusing on critical events such as heart failure [START_REF] Pakhomov | Prospective recruitment of patients with congestive heart failure using an ad-hoc binary classifier[END_REF] or adverse drug interactions [START_REF] Aspden | Generalized random shapelet forests[END_REF][START_REF] Ouchi | Machine learning to predict, detect, and intervene older adults vulnerable for adverse drug events in the emergency department[END_REF]. On the other hand, descriptive analytics approaches focusing on frequent pattern mining or subgroup discovery have been proposed.

More specifically, several ways of improving ADE detection have been explored [START_REF] Asker | Mining candidates for adverse drug interactions in electronic patient records[END_REF] by combining sequential pattern mining with disproportionality analysis. In particular, the use of sequential pattern mining for finding frequent sequences of drug event prescriptions have been explored, which then form the basis for the disproportionality analysis. In other words, instead of looking for unexpected drug-diagnosis pairs, the main focus is placed on extracting unexpected pairs of drug sequences and diagnoses. Since the proposed method is better suited to handle drug interactions, it is expected to handle cases where a sequential administration of interacting drugs is responsible for a certain ADE. An empirical investigation of the method has been performed using a subset of the Stockholm EPR corpus [START_REF] Dalianis | Stockholm epr corpus: A clinical database used to improve health care[END_REF].

In this paper, we focus on unsupervised learning, and more specifically on the detection of cluster structure in a medical database. Specifically, we use a large database of new-born babies treated at the neonatal intensive care unit [START_REF] Rinta-Koski | Machine learning in neonatal intensive care[END_REF] in Helsinki Children's Hospital in Finland. We use the data in retrospect to explore and investigate the diagnostic profiles of patients. For this aim, use use a clustering model to group the patient database to distinct patient groups, each having a particular diagnosis signature with the recorded diagnoses. We aim at identifying interpretable diagnostic profiles by characterizing the patient profiles by the most common diagnoses in the clusters.

In the rest of the paper, we describe some backgrounds of the origins of data and present related work in Section 2. The methodology and the experimental part of clustering the diagnostic profiles is presented in Section 3. In Section 4, we summarize our findings and conclude our paper.

Patient data and diagnoses as profiles

The data set under study has been recorded in the Helsinki University Hospital Neonatal Intensive Care Unit (NICU) between the years 1999 and 2013. The data set in question consists of some 2000 preterm babies born in the hospital and treated in the NICU. The treatment in the intensive care unit results in a lot of data recordings that can be analyzed in retrospect. Of particular interest in the data set has been the so called very low birth weight (VLBW) infants, which by definition are babies with a birth weight of less than 1500 grams. The statistics of these data are presented in more detail in [START_REF] Rinta-Koski | Machine learning in neonatal intensive care[END_REF]. In our previous work, we have explored the possibility to predict diagnoses based on the vital signs and other symptom data, based on Gaussian process classification [START_REF] Rinta-Koski | Gaussian process classification for prediction of in-hospital mortality among preterm infants[END_REF].

Contrary to our previous work [START_REF] Rinta-Koski | Machine learning in neonatal intensive care[END_REF][START_REF] Rinta-Koski | Gaussian process classification for prediction of in-hospital mortality among preterm infants[END_REF], where diagnoses were estimated or predicted from vital signs and symptom based data, we consider a different approach: we treat the diagnoses of NICU patients as an individual data resource and analyze the heterogeneity and the statistical dependencies within the data and the hypothesized groups in the data. For each patient, there is a list of diagnoses given to a patient during the NICU stay. We have extracted a list of diagnosed and some other variables and cross-referenced them with an ICD-10 database retrieved from a health organization THL. For our diagnostic profile, we include only those variables which are found in a standard ICD-10 database. As a result, we get a list of 437 possible diagnoses which have occurred in this data set. We must note that this is not a standardized, comprehensive list of all diagnoses but is focused on this set of patients specifically. As the list of diagnoses for a patient may vary, we seek to represent them as a unified diagnostic profile. Therefore, we represent the diagnoses for an individual patient as a list of truth values, which can be numerically represented as 0's and 1's. Our vectorial data representation has the possible diagnoses as attributes, and the binary 0-1 values denote whether a particular patient has a diagnosis [START_REF] Asker | Mining candidates for adverse drug interactions in electronic patient records[END_REF] or not (0). In this manner, we can represent the diagnoses as a binary 0-1 vector for a patient. This gives rise to a matrix where each row of the matrix represents the diagnostic profile of a patient.

Experiments: methodology and evaluation

Recall that our data is 0-1 data collected to a vectorial representation, where each vector describes the set of diagnoses for that patient with a collection of 0's (no diagnosis) and 1's (diagnosis) for a particular code in ICD-10. The analysis may now proceed as the analysis of multivariate 0-1 data. If we would wish to describe occurrences of diagnoses together, we could extract frequent itemsets from the 0-1 data [START_REF] Hand | Principles of Data Mining. Adaptive Computation and Machine Learning Series[END_REF], or extract frequent itemsets combined with a clustering approach [START_REF] Hollmén | Mixture models and frequent sets: combining global and local methods for 0-1 data[END_REF]. Here, we are doing the first steps in exploring the data and we are content by exploring the clustering structure with a clustering approach only.

During the experiments, the goal is to use the data of the diagnostic profiles described earlier and to learn a cluster model from data. For this aim, we use the k-means cluster model, where the cluster profiles are represented in terms of prototype vectors, computer locally from the clustered data [START_REF] Hand | Principles of Data Mining. Adaptive Computation and Machine Learning Series[END_REF]. A central question regarding the clustering is to choose the number of clusters in the model: an optimal choice of the model is a trade-off between the richness of representation (many clusters) and the compactness of representation (just a few clusters). As a guiding criterion, we use the silhouette index [START_REF] Rousseeuw | Silhouettes: A graphical aid to the interpretation and validation of cluster analysis[END_REF] computed from the clustered data and decide on the number of clusters based on a series of silhouette indices computed on the cluster model. In order to reduce the impact of individual cluster models learned from data, we compute multiple cluster solutions form different initial values and form averages of our chosen model selection criteria. We present the statistics of these figures for solutions between 2 clusters and 20 clusters. The results of the experiment for model selection is illustrated in Fig 1.

The choice of the number of clusters is a trade-off between the richness of representation and the compactness of the result. Whereas the compactness would make the result set very interpretable, the richness of the result would make the clustered data sets very homogeneous. In order to balance between these extremes, we resort to the silhouette index [START_REF] Rousseeuw | Silhouettes: A graphical aid to the interpretation and validation of cluster analysis[END_REF] and select the number of clusters to be J = 3. We observe declining silhouette score between 2 and 3 clusters, but an increase of clusters from 3 does not seem to affect the silhouette score. There is, however, some variance in the score, indicated by the variance in the individual scores marked by black points as well as the bounds given by the percentiles, marked by the dash-dotted lines. We proceed to the final clustering by fixing the number of clusters to be J = 3 and training a final model from data. In order to avoid degenerate results, we train a model 7 times and select the model with the median silhouette score. This is likely to avoid minima with extreme values for the silhouette index.

Since the k-means algorithm estimates the cluster centers as the averages of data and the data is either 0's or 1's, the cluster centers have a natural interpretation of being probabilities of individual diagnoses given in the cluster, and can thus be related with the risk of a diagnosis in a given group. The cluster centers for the three clusters are illustrated in Figure 2. There are apparent similarities between the profiles and they do indeed share some characteristics in terms of diagnoses. For instance, each cluster is characterised by the diagnosis P59.0, which indicates neonatal jaundice associated with preterm delivery, and P22.9, which in turn corresponds to unspecified respiratory distress of newborn. These are diagnoses that are associated with the selection of the patient material from NICU, rather than distinguishing factors between them. Some of the diagnoses appear in only one cluster, like the H35.1 Retinopathy of maturity, which of- fers an avenue to explore further which factors occur together in this particular group.

Summary and Conclusions

We have analyzed a database of patients treated at the neonatal intensive care unit (NICU) at the Helsinki Children's Hospital in Finland. In particular, we focused on the set of diagnoses of patients and developed a vectorial 0-1 data representation for further analysis. The diagnostic profile for a patient is the listing of all diagnosis of a patient and can be represented as a vector of 0-1 data with all diagnoses as vector components, or attributes. Then we proceeded with a clustering approach and developed a suitable clustering model for the data through a model selection procedure. We presented the prototypes of the clusters and discussed the further possibilities of describing the data more accurately.

A clustering model can be used to yield a practical, yet nontrivial description of the patient diagnoses as such. Some of the highlighted diagnoses are generic hallmarks for the patient material in question, but some others may yield interesting information about subsets of patients. These diagnostic profiles can be used to describe further the statistical dependencies of the individual diagnoses in subsets of patients, which can yield interesting, but unexplored knowledge about the domain. In order to derive more medical relevance from the profiles, we will discuss the findings further with the medical experts, and reflect the findings with external patient data, which has not been used in clustering.

Fig. 1 .

 1 Fig.1. The silhouette indices computed for multiple realizations of cluster models and number of clusters. We have clustered all the data in the patient matrix, with data dimension d = 437. We have run k-means clustering multiple times with random initializations, repeating the runs from 2 to 20 clusters 50 times. For each clustering result, we have computed the silhouette score for the solution at hand. Average of the scores calculated and plotted with a black line. Percentiles (25th and 75th) are plotted with dash dotted line. The individual scores are plotted with points.

Fig. 2 .

 2 Fig. 2. The cluster centers of the identified diagnosis profiles are illustrated. Each panel describes one of the three clusters in terms of probabilities of diagnoses in the cluster.

Acknowledgments

This work was partly supported by the VR-2016-03372 Swedish Research Council Starting Grant. The study was approved by the Helsinki University Central Hospital Ethics Committee, decision number 115/13/03/00/14 dated 8 April 2014. We thank Olli-Pekka Rinta-Koski for technical assistance with the data extraction.


Harald Schallner
email: harald.schallner@jade-hs.de

Capacity Requirements Planning for Production Companies Using Deep Reinforcement Learning Use Case for Deep Planning Methodology (DPM)

Keywords: Artificial Intelligence Applications, Planning and Resource Management, Deep Learning Framework, Deep Planning Methodology (DPM)

. The proposed deep neuronal network learns a policy choosing the best shift schedule, which provides the required capacity for producing orders in time, with high capacity utilization, minimized stock and a short throughput time. The deep learning framework TensorFlow TM [3] was used to implement the capacity requirements planning application for a production company.

Introduction

Many production companies have to plan the capacity requirements for their work center frequently. They have to decide how many working hours and shifts per day are needed for each production resource. Current standard software implementations of SAP ® Enterprise Resource Planning (ERP) and Supply Chain Management (SCM) systems support the capacity requirements planning by reporting capacity utilization and by capacity levelling functions based on shift schedule [START_REF] Gulyássy | Kapazitätsplanung mit SAP ®[END_REF]. Users have to maintain a feasible shift schedule before capacity requirements planning can be executed. SAP ® ERP and SCM systems do not offer any planning functionality to optimize shift schedule automatically [START_REF] Dickersbach | Supply Chain Management with APO: Structures, Modelling Approaches and Implementation Pecularities[END_REF]. One main reason for this missing standard system functionality is the significant uncertainty about changes of available production capaci-ties and customer requirements in the near future. Another reason is the exponential runtime complexity for a complete enumeration algorithm, which searches for the optimal solution within an exponential sized decision space. See O( ) -formula [START_REF] Sutton | Reinforcement learning: an introduction[END_REF] for complexity calculated by number of alternative shifts A (e.g. 8, 16 or 24 working hours per day), number of scheduled days T and number of capacities C:

𝑂 𝐴 ( 1 )
In order to achieve long-term goals for a sequence of inherent uncertain planning situations, reinforcement-learning approach is proposed in this work. Reinforcementlearning applications provide evaluative feedback to a learning agent interacting with an environment. This approach was chosen, because there were many reinforcementlearning implementations that could handle other decision problems with similar complexity and significant uncertainty successfully. Related work can be found in [START_REF] Sutton | Reinforcement learning: an introduction[END_REF].

Methodology for Applying Deep Reinforcement Learning to Planning Processes

Subsequent sections describe all steps that are proposed to implement deep reinforcement learning application for a planning process, named Deep Planning Methodology (DPM).

Fig. 1. Generic business process model

Step 1: Model Business Process

For the purpose of understanding the business process, data flow, used applications and the sequence of planning, process steps have to be modelled. The model has to comprise all relevant aspects of the business domain. Fig. 1 shows the graphical representation for all generic elements of the proposed business process model. For all planning process steps, the following business objects have to be modelled: All data objects have to be specified by their attributes, which are provided for or created by a planning process step.  Application systems:

Every application system has to be modelled, that supports planning functions or provides data objects.

Step 2: Identify Relevant Planning Process Steps

Every planning process step can be analyzed according to its relevance for applying deep reinforcement learning based on the business rules, the planner performance and the planning problem complexity. The following checklist helps to decide which planning process step has high potential for implementing: First check: If all business rules can be specified in detail, this means every organizational constraint can be formulated, then perform the second check. Else, go on with third check.

Fig. 2. Checklist for deep reinforcement learning applications

Second check: Is there any algorithm, heuristic or system transaction available, that computes a satisfactory planning result according to the objective function? If yes, the algorithm, heuristic or system transaction has to be analyzed in the fourth check. If there is no appropriate application available, then human planning skills have to be focused on in check three.

Third check: Find a human planner, who can create satisfactory planning results complying with implicit business rules manually. If there is nobody known with this competence, then go on with check four.

Fourth check: Analyze the computational complexity of the algorithm, heuristic or system transaction. If the runtime grows more than polynomially with the planning input size or the complexity is unknown, then deep reinforcement learning could be suitable for computing satisfactory planning results.

Generally, deep reinforcement learning applications are recommended, if the planning problem complexity is high or unknown or business rules cannot be specified completely and there is no planner performing satisfactory results, as shown in Fig. 2.

Step 3: Convert Planning Process into Markov Decision Process

In previous step two, planning process steps are identified that do not apply any algorithm with polynomial runtime. Fortunately, Papadimitriou and Tsitsiklis [START_REF] Papadimitriou | The complexity of markov decision processes[END_REF] proved that the Markov decision process for finite horizons is solvable in polynomial runtime by dynamic programming (so-called "P-complete"). Consequently, the planning process has to be converted into a Markov decision process.

In contrast to other machine learning, evolutionary or optimization approaches, the Markov decision process does not need exemplary supervision nor complete models of the business rules [START_REF] Sutton | Reinforcement learning: an introduction[END_REF]. The task of the Markov decision process is to compute an appropriate policy within an uncertain environment in order to achieve long-term goals at the end of a given time horizon. A policy decides which action a t has to be executed based on the current time t and state s t . Thus, planning process has to be defined as a sequence of state dependent actions for each time of a finite horizon 𝑇 ∈ 𝛮. The sequence of actions and states has to be modelled by time buckets 0 𝑡 𝑇, e.g. weeks, days or hours. In addition, a finite set of states S, an initial state s 0 for the first and for all other time bucket s t have to be specified. The state s t describes the decision relevant planning information for the current time bucket, provided by the planning environment. Furthermore, the action space has to be finite. Finally, a reward function r (s t , a t , t) ∈ ℝ has to be specified. To sum up, the Markov decision process has to find an optimal policy function 𝛿 𝑠 , 𝑡 , that maximizes the following value function v, as shown in [START_REF] Papadimitriou | The complexity of markov decision processes[END_REF]:

𝑣 ∑ 𝑟 𝑠 , 𝛿 𝑠 , 𝑡 , 𝑡 ( 2 )

Step 4: Develop Reinforcement Learning Environment

This methodology proposes the use of the deep learning framework TensorFlow that was developed as an open-source software library by the Google Brain Team and is available for everyone via GitHub [START_REF]TensorFlow Documentation[END_REF][START_REF] Géron | Hands-On Machine Learning with Scikit-Learn and TensorFlow[END_REF]. In recent years, TensorFlow has become very popular with over a million source code downloads [START_REF] Abadi | TensorFlow: A System for Large-Scale Machine Learning[END_REF]. In addition to Google, many companies have selected TensorFlow for their machine learning applications, e.g. Airbus, eBay, NVIDIA, Coca-Cola. Especially, SAP has integrated TensorFlow into its Leonardo Machine Learning Foundation by providing the "Bing your own Model" (BYOM) web service [START_REF] Leukert | Das intelligente Unternehmen: Maschinelles Lernen mit SAP zielgerichtet einsetzen[END_REF].

Based on TensorFlow Kuhnle, Schaarschmidt, and Fricke [START_REF] Kuhnle | Tensorforce: a TensorFlow library for applied reinforcement learning[END_REF] developed an open source library for applied deep reinforcement learning, named TensorForce. Tensor-Force offers a unified declarative interface to common reinforcement-learning algorithms [START_REF] Schaarschmidt | LIFT: Reinforcement Learning in Computer Systems by Learning from Demonstrations[END_REF]. TensorForce library provides an application-programming interface by the four following generic Python classes: Environment, Runner, Agent and Model [11].

The Agent class processes states, returns actions, stores past observations, loads and saves models. Each Agent class employs a Model class, that implements algorithms for calculating next action given the current state and for updating the model parameters from past experiences [11]. The Runner class implements the interaction between the Environment and the Agent. For every time step the Runner receives the current action from the Agent, executes this action in the Environment and passes the observation to the Agent. The Runner manages the duration and number of each learning episode and the cumulative rewards [11]. The planning horizon limits the number of time steps of each episode. The value function is implemented by cumulative rewards. Thus, the state sequence of one episode represents the planning process results for the time horizon. For each planning process step, following Environment methods and attributes have to be developed in Python [11]:

 Attribute actions returns the action space for all possible planning decisions.  Attribute states returns the state space for all possible planning situations.  Constructor method _init_() initializes the state based on planning input.  Method reset() sets up a new learning episode with its initial state for the first time step.  Method execute(actions) performs the selected actions and returns its reward, next state and a terminal indicator defining the end of the planning horizon.

TensorForce has no restriction on number and type of different states and actions [START_REF] Schaarschmidt | LIFT: Reinforcement Learning in Computer Systems by Learning from Demonstrations[END_REF]. To sum up, a planning specific Python class has to be developed, which overrides attributes and methods from the Environment superclass. A subclass has to simulate the consequences of each planning decisions to its domain. In order to handle the risk of unrealistic assumptions and oversimplifying, relevant business constrains have to be considered in the execute(actions) method.

Step 5: Configure Reinforcement Learning Agent

TensorForce library offers eight different pre-built Agent classes that implement state-of-the-art reinforcement-learning algorithms [START_REF] Schaarschmidt | LIFT: Reinforcement Learning in Computer Systems by Learning from Demonstrations[END_REF]:

 DQNAgent: Deep Q-Network agent applies the original Q-Learning algorithm [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF].  NAFAgent: Normalized Advantage Function agent uses continuous Q-learning algorithm [START_REF] Gu | Continuous Deep Q-Learning with Modelbased Acceleration[END_REF].

 DQFDAgent: Double Q-learning from demonstration agent enriches expert knowledge [START_REF] Van Hasselt | Deep Reinforcement Learning with Double Qlearning[END_REF].  VPGAgent: Classic Vanilla policy gradient agent implements the classic policy gradients algorithm, known as REINFORCE [START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF].  TRPOAgent: Trust Region Policy Optimization agent supports categorical, bounded and continuous action spaces [START_REF] Schulman | Trust region policy optimization[END_REF].  PPOAgent: Proximal Policy Optimization agent applies an alternative policybased method [START_REF] Schulman | Proximal Policy Optimization Algorithms[END_REF].  DQNNstepAgent: The n-step Q-Learning agent performs asynchronous gradient descent for optimization [START_REF] Mnih | Asynchronous Methods for Deep Reinforcement Learning[END_REF].  DDPGAgent: Deep Deterministic Policy gradient agent supports continuous action domains [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF].

Agent classes have to be configured at least by neuronal network layers, exploration, action space, state space, learning algorithm parameters and update methods. There are three update methods for network weights: episode based, batch based, time-step based. The selection of an appropriate agent class and the declarative configuration of hyper-parameters is required, when creating an instance in Python. The best way to choose a suitably configured class is to evaluate all eight classes experimentally, because each planning process step has different business rules, objective functions and planning data (see next step in following subsection 2.6).

Step 6: Evaluate Learning Results

The experimental evaluation of the configured agent classes interacting with the developed environment class requires representative input data. Therefore, data cleansing of real world planning input data is recommended, in order to create meaningful test scenarios. In addition, TensorBoard can be used as a dashboard for the deep learning models, implemented with TensorFlow [START_REF] Kanit | Visualizing Dataflow Graphs of Learning Models in TensorFlow[END_REF]. TensorBoard is an open-source implementation and available via GitHub [START_REF]The Tensor Board repository on GitHub[END_REF]. It generates web-based visualizations of the computation dataflow graphs and training curves, which enable to fine-tune hyper-parameters.

3

Use Case for Deep Planning Methodology (DPM)

Standard Business Process Model for Production Companies

In this subsection, the described Deep Planning Methodology (DPM) is applied to the SAP ® standard business process "plan-to-produce" for production companies. This standard planning process comprises the following consecutive planning steps [START_REF] Dickersbach | Supply Chain Management with APO: Structures, Modelling Approaches and Implementation Pecularities[END_REF]: Demand Planning consolidates the sales forecast, applies statistical methods and releases a demand plan to Production Program Planning. The goal of DP is to achieve a high forecast accuracy. Based on demand plan, sales orders and delivery schedule, PP calculates balanced production quantities for every finished product and period. PP's objective comprises the on-time delivery performance and the probability that demand is met from stock, called service-level. PP plans the requirement quantity and dates for finished products and releases independent demand to MRP. Subsequently, MRP creates planned orders for each in-house production material, in order to meet the independent demands. A planned order schedules dependent demand for its components and a technical sequence of operations performed at capacities. These operations require adequate capacity availability for every scheduled operation. The duration of an operation defines the capacity requirement quantity. MRP assumes infinite capacities. This means, MRP does not check scheduled operations with regard to capacity availability. Thus, the availability of capacities has to be checked and ensured by the following Capacity Requirements Planning (CRP). Based on planned orders CRP has to plan feasible shift schedules for all production capacities. On the other hand, Detailed Scheduling (DS) has to consider the material availability of all components based on shift schedule and planned orders. DS schedules the sequence of operations performed at capacities and aims to minimize set-up efforts.

1.

Capacity Requirements Planning Process

Fig. 3 visualizes the CRP process with its business rules, objectives, application systems and data objects. The shift schedule can be modeled by the decision variables d i,j (C x T -matrix) that defines the available working hours for all capacities 𝐶 ∈ ℕ and each day within a defined planning horizon 𝑇 ∈ ℕ:

𝑑 ∈ ℝ , 0 𝑑 , 24 (3)
A planned order is specified by its operations i:  Operation sequence: 𝑖 ↠ 𝑗 ⇔ i is predecessor of operation j  Scheduled time bucket: 0 𝜏 𝑇  Production duration:

𝜚 ∈ ℕ  Required capacity:

𝜓 ∈ ℕ Capacity requirements k can be summed by all relevant operation durations:

𝑘 ∈ ℝ , 𝑘 , ∑ 𝜚 ∀ : ∧ (4)
A planned shift schedule is feasible, if two business constrains are fulfilled: Firstly, finite capacity utilization is limited to 100% for all capacities c and time buckets t:

∀ 𝑐, 𝑡: 𝑈 , , , 100% ( 5 )
Secondly, the technical sequence of operation is ensured, if succeeding operations have to be scheduled at a later time bucket:

∀ 𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑜𝑛 𝑖 ↠ 𝑗 ∶ 𝜏 𝜏 (6)
The objective function f that measures the quality of a shift schedule comprises classic key performance indicators used in production companies, such as capacity utilization U c,t , weighted by w u , throughput time P i , weighted by w p , inventory R i , weighted by w r , and delivery performance P j , weighted by w p :

𝑓 𝑤 ∑ 𝑈 , ∀ , 𝑤 ∑ 𝑃 , ∀ , 𝑤 ∑ 𝑅 , ∀ , 𝑤 ∑ 𝑃 , ∀ , (7)
CRP was identified as a relevant planning step with high potential for deep reinforcement learning implementations, because the complete enumeration algorithm has exponential runtime complexity, see equation (1).

Markov Decision Process for Capacity Requirements Planning

CRP is converted into a Markov decision process by defining actions, states and rewards:  A C-dimensional vector specifies the actions that decide how many working hours (here: 8,16 or 24) are available for each capacity and time bucket t:

𝑎 ∈ 8, 16, 24 ( 8 )
 The shift schedule d is calculated by the actions a t performed for all capacities within one learning episode:

𝑎 𝑑 , ⋮ 𝑑 , ( 9 )
 State s t is designed via capacity requirements based on a time interval with b buckets. This means, that the state slices the capacity requirements by a rolling time window:

𝑠 𝑘 , ⋯ 𝑘 , ⋮ ⋱ ⋮ 𝑘 , ⋯ 𝑘 , (10)
 In orders to ensure, that for each learning episode the cumulative reward v is equal to the objective function value f, reward function r is calculated by:

𝑟 𝑠 , 𝑎 , 𝑡 𝑤 ∑ 𝑈 , ∀ 𝑤 ∑ 𝑃 , ∀ 𝑤 ∑ 𝑅 , ∀ 𝑤 ∑ 𝑃 , ∀ (11)

Learning Environment for Capacity Requirements Planning

The learning environment has to fulfill the two business constraints, which are described in subsection 3.2. In consequence, the execute(actions)method implements for each time step and each capacity that the planned shifts offer enough working hours to meet the capacity requirements. If capacity utilization is higher than 100%, operations are moved to the next time bucket. Else operations are pulled forward from subsequent to current time buckets, in order to achieve fully utilized capacities. Predecessors and successors of moved operations have to be rescheduled, too. This influences throughput time, inventory, delivery performance and consequently the reward. The implemented time window of states focuses on a rolling interval of two days for capacity requirements. This provides relevant information for the agent to find a satisfactory shift schedule.

Learning Agent for Shift Schedule

Thriving reinforcement learning often requires tuning the agent hyper-parameters. This subsection proposes some best practice for customizing of the selected PPOAgent class, which has proven its productivity in the domain of capacity requirements planning.

The hidden layers of deep neuronal network are specified by parameter network. State and action spaces define the structure of input and output layers implicitly. Explicitly, three hidden layers have to be configured to meet the complexity of this capacity-planning problem. The number of hidden units corresponds with the planning complexity, described in O -formula [START_REF] Sutton | Reinforcement learning: an introduction[END_REF]. This means high number of capacities, longer planning horizon and high number of alternative shifts require higher number of hidden units, called size of layer:

1. The first layer type is embedding, in order to handle all dimensions of the state space. This layer represents states in a continuous vector space where semantically similar states are mapped to nearby points, coded as indices. The number of embedding indices should be within the range of 32 to 1024 (here 100). The dimension of the vector space is defined by the size of the embedding layer, here 32 (range: 16 -1024). This means, that the input layer is fully connected to the embedding layer. L1 and L2 regularization weights are not used in this layer, because overfitting is not an issue for learned policies. 2. The second layer is called flatten. This layer reshapes the first layer from multidimensional vector to a one-dimension vector. 3. The third dense layer with 32 hidden units (range: 32 -1024) is fully connected to the second layer. The ReLU function [START_REF] Géron | Hands-On Machine Learning with Scikit-Learn and TensorFlow[END_REF] is chosen for activation.

Prominent results concerning the convergence of cumulative rewards to a satisfactory solution can be achieved by the following configuration of the PPOAgent class:

 The update_mode specifies how many episodes or time steps used for one iteration of a gradient descent update. The frequency defines the number of elapsed episodes or time steps that trigger the next update batch run. Both variables are set to one episode (range: 1 -20 episodes).  The capacity of the memory corresponds to the numbers of experiences collected before updating neuronal network weights. It has to be a multiple of the up-date_mode batch size and was set to 100000, to speed-up training.  The learning rate specifies the strength of each gradient descent update step. Best results for Adam optimization were achieved with rate 0.001 (range: 0.00001 -0.008). If scattering of the objective function values is too high or the rewards do not consistently increase for many episodes, the learning rate has to be decreased. On the other hand, if the learning rate is too low, then optimal solution is too hard to find for the agent.  Parameter entropy_regularization controls the randomness rate of the policy. This parameter is very sensitive and should have the value 0.03 of interval 0.001 to 0.1. Other values have negative effects on convergence and level of cumulative rewards. Higher values create instable solutions for many periods. Lower values prevent finding satisfactory solutions.  Parameter likelihood_ratio_clipping specifies the acceptable threshold of divergence between the new and old policies for gradient update step. High values correspond with a fast, but instable learning process. Best results are based on value 0.02 (range: 0.005 -0.35).  Parameter discount defines the factor of future rewards. The parameter value 0.99 ensures policies with high cumulative rewards at the end of an episode. Thus, discount factor should have a high value within the interval of 0.85 to 0.999, because the objective function of capacity requirements planning is calculated by the cumulative rewards for each time horizon.

Experimental Evaluation

Four of eight TensorForce agent classes are suitable for CRP: DQNAgent, VPGAgent, PPOAgent and DQNNstepAgent. The other agents were not applicable, because they are designed for continuous action space or additional expert knowledge. The qualified agents were evaluated according to ten different test scenarios. The test scenarios differ according to randomized planned orders, number of ca-pacities (range: 6 -24), number of operations (interval: 2.500 -10.000) and planning horizon from 30 to 120 days. Learning algorithms were finished after 4000 episodes.

Conclusion and Future Work

The proposed Deep Planning Methodology is an advanced implementation paradigm for planning processes in companies. It allows satisfactory planning results by developing a learning environment and customizing an agent without the need to implement all business rules and optimization algorithms explicitly. In the course of the CRP implementation a challenge emerged, how to tune hyper-parameter. Some hyper-parameters are sensitive concerning convergence and level of planning results. To meet the challenge many test scenarios are recommended to get a comprehensive insight, how to achieve reliable and reproducible results. To sum up, it is fascinating, how complex decision problems can be solved by reinforcement learning algorithms.

In the future, I expect more Eureka moments for other planning tasks in companies, which have been unable to be successfully solved by current business applications until today.

Fig. 3 .

 3 Fig. 3. Capacity Requirements Planning Process Model Evaluation criteria are the convergence and level of the weighted objective function. Consequently, for each test scenario, each agent configuration and all training episodes the cumulative reward values have to be compared. In addition, TensorBoard can be used as a dashboard for the deep learning models, implemented with TensorFlow [20]. TensorBoard is an open-source implementation

  All test scenarios converge to a satisfactory result, near global optimum. The mean of cumulative rewards constantly increased over periods. The scattering of the objective function values decreased over periods. The runtimes were reasonably short: from 0.3 to 1.2 seconds per episode. TensorFlow ran with the GPUs of NVIDIA Quadro M1200 with a compute capability of grade 5.0. The PPOAgent performed best results according to level and convergence of the objective function, as show in Fig. 4.

Fig. 4 .

 4 Fig. 4. : Evaluation of test scenario with six capacities, 2500 operations and 30 days horizon


Ernst-Josef Behmer
email: ernst-josef.behmer@iosb.fraunhofer.de

Krishna Chandramouli
email: k.chandramouli@venaka.co.uk

Victor Garrido

Dirk Mühlenberg
email: dirk.muehlenberg@iosb.fraunhofer.de

Dennis Müller
email: dennis.mueller@iosb.fraunhofer.de

Wilmuth Müller
email: wilmuth.mueller@iosb.fraunhofer.de

Dirk Pallmer
email: dirk.pallmer@iosb.fraunhofer.de

Francisco J Pérez

Tomas Piatrik
email: t.piatrik@qmul.ac.uk

Camilo Vargas

Ontology population framework of MAGNETO for instantiating heterogeneous forensic data modalities

Keywords: Information Extraction, Ontology Population, Law Enforcement

The growth in digital technologies has influenced three characteristics of information namely the volume, the modality and the frequency. As the amount of information generated by individuals increases, there is a critical need for the Law Enforcement Agencies to exploit all available resources to effectively carry out criminal investigation. Addressing the increasing challenges in handling the large amount of diversified media modalities generated at high-frequency, the paper outlines a systematic approach adopted for the processing and extraction of semantic concepts formalized to assist criminal investigations. The novelty of the proposed framework relies on the semantic processing of heterogeneous data sources including audio-visual footage, speech-to-text, text mining, suspect tracking and identification using distinctive region or pattern. Information extraction from textual data, machine-translated into English from various European languages, uses semantic role labeling. All extracted information is stored in one unifying system based on an ontology developed specifically for this task. The described technologies will be implemented in the Multimedia Analysis and correlation enGine for orgaNised crime prEvention and invesTiga-tiOn (MAGNETO).

Introduction

The Multimedia Analysis and correlation enGine for orgaNised crime prEvention and invesTigatiOn (MAGNETO) formalizes a structural framework to enhance the operational capability of Law Enforcement Agencies (LEAs) in their fight against organized crime and terrorist organizations. Following the recent terrorist attacks reported across Europe in London, Paris, Berlin, Brussels, the critical challenges faced by respective investigative agencies include the processing of large volumes, the heterogeneity and the fragmentation of the data that officers have to analyze for the prevention, investigation and prosecution of criminal offences. This paper contains the first stage of MAGNETO, co-funded by the European Commission within the "Horizon 2020" program. The overall integrated framework is presented in Fig. 1. As the overall investigative framework brings together several research components, the paper highlights the research carried out in "heterogeneous data mining and evidence collection" which is interfaced with the high-level semantic reasoning component.

Fig. 1. Proposed framework of MAGNETO

The first step to make use of the available data is the extraction of the contained information and its representation in a machine-readable way. Due to the diversity in the origin of the different evidences, e.g. audio, video, pictures and textual data, different methods for the extraction of information are necessary. For multimedia content indexing, background/foreground subtraction provides the separation of a video stream into foreground, which is characterized by its appearance at unique moments in time, and into omnipresent background. For indexing audio streams, a speech to text module will be supplied for transforming the speech contained in audio files into text. Furthermore, the Distinctive Region or Pattern (DROP) detection module allows the retrieval of videos in large databases containing a predefined pattern of distinctive visual effect. The text mining module extracts information contained in English text data composed in natural language. Text data which is not in English language requires a translation engine to translate selected European languages to English. All available information will be stored using an ontology designed for the specific needs of the MAGNETO project.

In Section 2, we describe approaches and implementations of the different information extraction methods whereas Section 3 is devoted to the underlying ontology and its population.

2

Information Extraction Methods

Multimedia Content Indexing

In this section, we give an overview of multimedia content indexing solutions implemented in MAGNETO for analyzing heterogeneous data in the process of evidence collection.

Foreground/background subtraction

Foreground/background subtraction can be defined as a segmentation of a video stream into foreground, which appears at unique moments in time, and the background, which is always present. It is an important video processing task that contributes to higher precision and accuracy when integrated with other video analytics solutions for forensic evidence collection.

Problem Definition

Vast range of video analysis tasks often begins with background subtraction, which consists of creating a background model that allows distinguishing foreground pixels. Processing per-pixel basis from the background is not only time-consuming but can also dramatically affect foreground region detection, if region cohesion and contiguity is not considered in the model. Decomposition of a video scene into background and foreground is a very challenging but crucial process that can significantly improve performance of many other video analytics solutions. The robust subspace approach based on a low-rank plus sparse matrix decomposition has shown a great ability to identify static parts from moving objects in video sequences. However, those models are still insufficient in realistic environments. The separation of locally moving or deforming image areas from static or globally moving background is a focused video-processing task with manifold applications.

Approach and configuration used for MAGNETO

In MAGNETO we implement a new method in which we regard the image sequence to be made up of the sum of a low-rank background matrix and a dynamic tree-structured sparse matrix, and solve the decomposition using an approximated Robust Principal Component Analysis (RPCA) method extended to handle camera motion, see [START_REF] Erfanian Ebadi | Efficient background subtraction with low-rank and sparse matrix decomposition[END_REF]. Furthermore, to reduce the curse of dimensionality and scale, a low-rank background modelling is integrated via Column Subset Selection [START_REF] Boutsidis | An Improved Approximation Algorithm for the Column Subset Selection Problem[END_REF] that reduces the order of complexity, decreases computation time, and eliminates the huge storage need for large videos. The proposed approach builds on the findings accomplished by Candès et al. [START_REF] Candes | Robust principal component analysis?[END_REF], where the authors provided a practical solution for the long-standing problem of recovering the low-rank and sparse parts of a large matrix made up of the sum of these two components. The background part of the video sequence is modelled by the lowrank matrix, while the locally deforming parts constitute the sparse matrix component. Although the approach described by Candès et al. [START_REF] Candes | Robust principal component analysis?[END_REF] leads to a computationally feasible solution, the complexity is still high involving the calculation of many Singular Value Decompositions (SVD) for a very large matrix.

The proposed solution implemented in MAGNETO addresses a number of critical issues and limitations of RPCA which are: embedding global motion parameters in the model, i.e. estimation of global motion parameters simultaneously with the foreground/background separation task; considering matrix block-sparsity rather than generic matrix sparsity as natural feature in video processing applications; and more critically providing an extremely efficient algorithm to solve the low-rank/sparse decomposition task. The first model aims at video sequences captured by a moving camera, by estimating the global motion parameters while performing the targeted background/foreground separation task. The second model exploits the fact that in video processing applications the sparse matrix has a very special structure. In other words, the non-zero matrix entries are not randomly distributed, but they build small blocks within the sparse matrix. Finally, the last solution targets the fact that RPCA approaches are computationally expensive. The proposed model introduces an extremely efficient "SVD-free" technique that enables real-time foreground separation.

Fig. 2. Results of foreground/background subtraction (pictures in second row represent extracted foreground objects)

The implemented method can handle camera movement, various foreground object sizes, and slow-moving foreground pixels as well as sudden and gradual illumination changes in a scene. The foreground/background subtraction (see Fig. 2) serves as an important pre-processing step for other video analytics modules within the MAGNETO system.

DROP detection and tracking

As the volume of information to be processed and analyzed increases exponentially, the suspect identification becomes an ever-challenging activity for the LEAs. The problem is compounded by the fact that potential suspects tend to mask their face during the act of violence. In addition, the limited resolution of the largely deployed CCTV network across several locations limits the ability of officers to successfully identify suspects. Addressing these challenges, MAGENTO platform integrates, the Distinctive Region or Pattern (DROP) [START_REF] Mitrea | Retrieval of Distinctive Regions of Interest from Video Surveillance Footage: A Real Use Case Study[END_REF] module that relies on the visually distinctive region as identified by the investigators to guide the search across large datasets of CCTV footage. This query region is then used as a query by example for the search of similar patterns across the large databases of CCTV footages.

Problem Definition

To illustrate a case of DROP based orchestration, a large corpus of CCTV and mobile data was used to identify an offender in a large city. The four images in Fig. 3 outline a common case of subject identification performed by security forces. In this case the distinctive "orange zipper" (DROP) in the black jacket of the perpetrator (attacking another individual in the first image), can be used for person identification in subsequent images. Therefore, the first image illustrating the criminal act is used as starting point for the investigation with the investigator identifying the DROP to be used to filter the repository content. The next two images depict the results of the content retrieval where standardized DROP metadata is used to search the media repository. Finally, the image at the right displays a picture containing the same DROP in which the face of the perpetrator can be clearly identified. It is expected that the DROP based media retrieval contributes to solve similar and more complex cases by performing automated searches of standardized metadata in all images and videos available for case solving.

Approach and configuration used for MAGNETO

The DROP module combines a machine learning approach with a tailored cascade classification technique. The machine learning part is based on a random forest of decision trees, which is trained with the content of the video to be searched. Traditional methods train a forest with exemplars of the pattern or object class to be detected. By training the forest with the haystack, rather than the needle, it can be reused to detect any pattern that the user may choose. The cascade classification approach involves two stages: definition and search. Firstly, the target DROP is described using color and Haar-like features [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] in order to define the conditions to be matched in the search stage. Subsequently, a sliding-window based search stage is performed in the target images. The search space in the image is initially reduced by defining an attentional grid using a dominant color matching approach. Then, for each possible sliding-window in the attentional grid a set of conditions is assessed in order to compare the current region against the query pattern. If every condition is satisfied for a specific sliding-window, the region covered by the window is marked as a positive detection. The steps involved are highlighted in Fig. 4.

Fig. 4. -DROP search process

The extracted DROPs are indexed as bounding box coordinates to be subsequently used by the investigators for the recognition of the potential perpetrators. Each pattern generated by the investigators is treated as an image region query for which a set of video segments from the CCTV dataset is retrieved. The spatio-temporal reasoning algorithms implemented on the MAGNETO knowledge repository will facilitate the filtering of irrelevant suspects and thus simplify the process of suspect identification.

Speech to text transformation

LEA investigators deal with enormous and complex amounts of data from calls or recorded audio files; it is impossible to listen all of them to discover hidden or unsuspected connections within big datasets. This provokes expensive investigations with enormous delays solving crimes and hard time to prevent them. Multilingualism conversations are hard to decrypt and usually data sources are not contextualized. Since MAGNETO's platform provides new semantic technologies and augmented intelligence tools for indexed textual contents, a speech to text module brings to MAGNETO a solution that combined with other modules improves investigation capabilities.

Problem Definition

Nowadays telephone tapping, recorded audio files from social networks, questionings or interrogations, among others, are suitable evidences that can be used in real use cases. The problem appears when LEAs have to listen and process all these audio files. Most of the records contain irrelevant information which implies a waste of time and resources from other conducted tasks or investigations.

Actual LEA's computer infrastructures are not able to manage the different algorithms based on machine learning used in audio processing techniques yet, high performance computing is needed.

Approach and configuration used for MAGNETO

The speech to text module provides to MAGNETO a powerful tool able to combine online conversions through the APIs exposed by the main providers (e.g. GOOGLE, MICROSOFT, IBM, WIT, among others) in case that internet access at LEA premises is allowed, or offline conversions based on local dictionaries in case of security policies or privacy restrictions, as shown in Fig. 5 The result obtained from MAGNETO's Speech to Text module is the audio file transcription. The file can be analyzed by other MAGNETO modules responsible for pattern identification, sentimental analysis or texts classification. This automated approach is clearly more feasible than extracting information by listening directly to the audio files. LEAs will be provided with a fast and robust solution also capable to interoperate among other platform functionalities.

Fig. 6. Speech to text module integration

To improve the performance of the speech to text module, it has been designed taking in to account the integration into MAGNETO's Big Data Services. The solution proposed in Fig. 6, uses the Hadoop Distributed File System (HDFS) as a repository for all the audio files to be transcribed. Once a file is uploaded to the repository, an Apache Spark job starts the conversion, and the result of this transformation is associated as metadata to the audio file and is stored in a NoSQL database for future analysis by other modules.

Text Mining

An ongoing investigation produces vast amounts of textual data, some of which is composed in natural language with all its vagueness and ambiguity. The purpose of the text mining module is to provide the methods and algorithms for retrieving high-quality information from these textual data, which then can be incorporated into the MAGNETO knowledge database.

Problem Definition

A core problem in deriving information from textual data is the identification of information fragments. Information fragments in this context are basic concepts as names, locations, organizations, time specification, but also relations between the former. In order to extract these fragments from the data, linguistic analysis of the textual data is necessary, which then clarifies the meaning of a given word in the context of the textual data.

Approach and configuration used for MAGNETO

Text mining, i.e. the retrieval of information from textual data, has possible applications in the mining of confiscated emails and documents confiscated during housesearches, transcribed telecom/audio data, and other documents used in an investigation.

The linguistic analysis includes e.g. part-of-speech tagging, semantic role labelling (shallow parsing), named entity recognition, and capturing the meaning of a sentence in natural language. All these rely on theoretical models used either in rule-based natural language processing (NLP) or in statistical NLP, which is based on algorithms from machine learning or on a new approach using deep neural networks.

The result of the processing is an annotated document from which the relevant information can be extracted in form of a conceptual graph of pre-processed forensic information. At last, the information contained in the conceptual graph needs to be incorporated into the ontology.

For extracting information from textual data, we employ the technique of Semantic Role Labelling using Pikes [START_REF] Corcoglioniti | A 2-phase frame-based knowledge extraction framework[END_REF] which tries to capture the meaning of a sentence in natural language. The goal is to represent the meaning with a rooted, directed, acyclic graph with labels on edges (relations) and leaves (concepts). Pikes performs traditional tasks such as named entity recognition, semantic role labelling and word sense disambiguation by using techniques like Probabilistic Graphical Models (PGM), including Bayesian Networks, Hidden Markov Models, or Conditional Random Fields.

Different frameworks are available which implement the various techniques. In the case of Pikes, among others the product called Stanford CoreNLP [START_REF] Manning | The Stanford CoreNLP Natural Language Processing Toolkit[END_REF], which is an object-oriented framework implemented in Java, is used.

An example of a graph generated by Pikes for the sentence "Lawyers for the survivors have filed a complaint against Sharon in Belgium" is given in Fig. 7. Fig. 7. An example Pikes graph. Source: http://pikes.fbk.eu/

Automatic Language Translation

This module translates text composed in predefined European languages to English.

Problem Definition MAGNETO's algorithms are configured for English language, while the information comes in different languages. The lack of an English version of the texts significantly slows down the investigation process.

A typical human translation is an expensive and long process. Nowadays, the amount of information increases constantly, and the use of human translation creates a bottleneck in the effectiveness of organizations. If the text volume ramps up to hundreds or thousands of words, it becomes nearly impossible to use human translation.

The majority of machine translation solutions available on the market is cloud based or have licensing limitations. In both cases, they are impossible for use in MAGNETO since it requires no severe licensing limitations and on-the-premise installation with no external connection to the internet.

Approach and configuration used for MAGNETO

The pursued approach is to train a proprietary engine based on available open-source code technology (e.g. OpenNMT [START_REF]OpenNMT Homepage[END_REF]) for statistical machine translations. The idea behind statistical machine translation comes from information theory. A document is translated according to the probability distribution that a string e in the target language (for example, English) is the translation of a string f in the source language (for example, French). Therefore, for the training of the engines large corpora of domain specific translations are needed. In order to ensure a high quality translation, the engines for MAGNETO will be trained with 1,000,000 bilingual segments.

MAGNETO Knowledge Base

The knowledge base consists of several databases for storing the raw datasets and an ontology [START_REF] Borst | Construction of Engineering Ontologies[END_REF], which summarizes the higher-level information, retrieved from the raw datasets. The latter is described in detail in this section.

The underlying Ontology

In MAGNETO, the knowledge modelling process is crucial to ensure the goal of developing a platform that represents all relevant investigational elements, integrates different and heterogeneous information, performs advanced computational analytics, generates recommendations and inferences, and assists LEAs with their criminal investigation. In order to effectively define knowledge representation, it is necessary to understand what an investigative process is. An investigation is a complex process that involves event, person, circumstance, various types of resources, data extracted from resources, scenarios and relations that link all these elements. The objective of an investigation is understand to who, why, how, by what means, when and where a crime was perpetrated in order to identify each role of all involved persons.

Fig. 8. MAGNETO ontology conceptualisation for criminal investigation

In [START_REF] Dragos | Developing a core ontology to improve military intelligence analysis[END_REF], the author describes a core ontology based on NATO standards to improve military intelligence analysis. The main concepts of this core ontology have been selected as a basis of the MAGNETO ontology: Organization, Equipment, Person, Place, and Event as depicted in Fig. 8(a). A more detailed outline of the Event category is presented in Fig. 8(b). Besides these concepts, further top-level concepts are required. The general concept PersonAttributes, for example, serves the purpose to describe the characteristics of a suspect or perpetrator. (Sub-)Sub-concepts for describing details are, e.g., FingerPrint, Clothing, or Tattoos. Lower levels of the ontology represent domain-specific concepts, e.g. Crimes, Arson, Thief, Victim, Middleman, Witness, on the one hand, and scenario-specific concepts on the other hand, e.g. HighSecurityLock, ArmouredCar, RealEstateInvestor, HeartAttack.

Fig. 9. Event hierarchy of criminal investigation

Exemplary parts of the further elaboration of the top level concept Event are depicted in Fig. 9, which is specifically tailored to address broadly the various criminal investigation processes and procedures. Similarly, each concept in the pentagram model is further elaborated. These concepts and their sub-concepts are associated among each other through object relationships. The involved LEAs will evaluate the MAGNETO platform and ontology using real investigation cases. This experience will be used to refine the ontology, which will then be published and submitted for standardization.

Population of the MAGNETO Ontology

The instantiation of the proposed ontology through metadata extracted from analytics components described in Chapter 2, presents a unique challenge. The technical implementation of the above approach is done as in the following: The instances representing the high-level information in a given investigation are stored in a RDF triplestore provided by the Apache Jena [12] framework. The information extracted from heterogeneous data sources is communicated to the triple-store via the interface implemented by an Apache Fuseki server. Ideally, the entire triple-store is held in the memory to guarantee computationally efficient runtime. However, nowadays real world cases have to process a vast amount of information preventing to keep everything in memory at the same time. A solution to this is the TDB2 storage concept of Apache Jena, which has theoretically no size limitations and allows 100's of millions of triples.

The task at hand in the population of the concepts is to decide which concepts of the ontology need to be instantiated to store the information extracted by the different information extraction modules. As a start, this may be done with lookup tables, which decode which instances are created as a result of incoming information.

In the context of natural language processing, the lookup tables are supplied by databases like DBpedia [START_REF] Mendes | DBpedia: A Multilingual Cross-domain Knowledge Base[END_REF] and SUMO [START_REF] Pease | The Suggested Upper Merged Ontology: A Large Ontology for the Semantic Web and its Applications[END_REF]. The use of DBpedia allows refining the type of the main concepts (see Section 3.1) Events are refined based on SUMO database. The found types need to be mapped to concepts modelled in the MAGNETO ontology. The mapping is implemented by determining the semantic similarity. For the similarity, a statistical probability measure based on word embedding [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] is used.

Conclusions and future work

In this paper, an outline of various information extraction and processing components is presented to aid the LEAs in their criminal investigation. The output of the information extraction components is used to populate an ontology which has been developed specifically for MAGNETO's purposes. In detail, the MAGNETO ontology concepts are instantiated according to the extracted information and stored by using the Apache Jena framework completed by an Apache Fuseki server for communicating with the knowledge repository. The future work will focus on the usage of instantiated information, e.g. temporal sequences, to support semantic reasoning of event occurrences leading to perpetrator identification.

Fig. 3 .

 3 Fig. 3. Identifying offenders by DROP based orchestration. Scene of the incident (image at the left), images of strong DROP correlation (middle), final image with the same drop and clear face of the perpetrator.

Fig. 5 .

 5 Fig. 5. Speech to text transformation flow diagram using CMUSphinx [6]