D Vijay Rao

Jasleen Kaur

A Fuzzy Rule-based approach to design game rules in a Mission Planning and Evaluation system

Keywords: Military simulation, Mission effectiveness, Rule-based systems, Damage Assessment Modeling, Fuzzy logic

Simulations and wargames offer powerful representations to model the mechanics and psychology of military operations that are inherently complex. They offer mechanisms to predict and assess the effectiveness of the mission plans and operations in achieving the military objectives. In this paper, we present a new approach to design the games rules of wargames using fuzzy rule bases, for quantitatively evaluating the effectiveness of air tasking missions. We determine the comparative damage relative to intended damage for a target, taking into account the effects of operational characteristics to compute possibilistic damage to the target as opposed to the probability of damage to the target. The cookie-cutter method to compute the damage is modeled as a fuzzy variable. Effectiveness of the mission is obtained by comparing the damage to targets with the cost and significance of the target in meeting the mission objectives. Damage assessment computation to targets using fuzzy rule bases gave more realistic results when used in field training and deployment of the system.

Introduction

Simulation has been applied extensively and successfully to a wide range of military problems, including wargaming, acquisition, logistics, and communications. It has been used as a decision support tool to evaluate how a battle force should be constituted, how to plan the force strength and structures, how it might be deployed, and how the weapon systems should be acquired and maintained. Wargames constitute an important class of applications that have proved to be an important tool for military system analyses and an inexpensive alternate to live training exercises. However, modeling the complexity and battle dynamics of the real world, assessing and predicting the outcomes of missions plans quantitatively and accurately is a very difficult endeavor [START_REF]Handbook of Simulation: Principles, Methodology, Advances, Applications, and Practice[END_REF], [START_REF] Vijayrao | The Design of Air Wargame System Simulation system[END_REF]. Air Wargame Simulation System (AWGSS) is a wargaming software that has been developed for planning and evaluating air tasking operations (missions) [START_REF] Vijayrao | The Design of Air Wargame System Simulation system[END_REF]. Given a military objective, the air component commander plans missions, minimizes the risk while targeting, chooses appropriate weapons for targeting while ensuring the intended damage is caused to the target. This is done considering the enemy forces' deployment, weather and terrain encountered en-route and suitability of aircraft, weapons for achieving the objectives. Traditionally, the occurrence of various events has been represented using probability distributions, a set of game rules that represent various situations and classical logic to compute the damage assessment. Damage assessments for targets were implemented using crisp cookie-cutter function that gives the probability of damage of a target. These methods are found to be inadequate in the present war scenario, to generate a realistic assessment of the mission plans due to uncertainty and vagueness of the factors considered. In this paper, we present a novel approach to design the games rules of wargames using fuzzy rule bases, for quantitatively evaluating the effectiveness of air tasking missions.

Fuzzy Rule-based Representation of Game Rules

Traditional approaches to wargame simulations use classical logic for damage assessment. Classical two-valued logic system, crisp set theory and crisp probability on which the damage assessment is based, are inadequate and insufficient for dealing with real-life war scenario that involves complexity and different sources of uncertainty. Damage assessment for a target done using cookie-cutter function gives the probability of damage of a target. Crisp cookie-cutter function states that a target is damaged inside a circle of specified radius r, and no damage occurs outside it [START_REF] Vijayrao | The Design of Air Wargame System Simulation system[END_REF].

d(x, y) = 1, x²+ y² ≤ r² 0, otherwise (1)
where, d(x, y) is the damage function of the point target by a weapon whose point of impact is (x, y). The target is assumed to be completely damaged within the circle of radius r and no damage occurs outside r. The notion of probability stems from, and depends on, the idea of repeated trials. Under identical and repeatable laboratory conditions conducted on simple models, this probabilistic notion readily applies; but, in real-world systems, experiments are rarely identical and repeatable. Therefore, for the subjective assessment of complex military systems, probability has its limitations. Game rules in conventional mission processing are limited and simplistic, taking into account limited number of parameters and based on statistical distributions. They do not take into consideration, effects of weather conditions, operational environment, target attributes, significance attached with the target, visibility of target at time of target acquisition, and effects of terrain while assessing damage. Variables such as these play a vital role in mission planning, mission processing and damage assessment. Different commanders deciding on the result of a planned mission are usually subjective based on their experiences. In order to overcome the limitations of statistically steady state results for game rules, we propose a fuzzy rule based approach for the design of the game rules. What is gained through fuzzification is greater generality, higher expressive power, an enhanced ability to model real-life problems, a methodology for exploiting the tolerance for imprecision and which serves to achieve tractability and robustness [START_REF] Cox | The Fuzzy Systems Handbook[END_REF], [START_REF] Mendel | Uncertain Rule-Based Fuzzy Logic Systems-Introduction and New Directions[END_REF]. Fuzzy Logic is the logic behind approximate reasoning instead of exact reasoning. As knowledge acquisition in wargames design and development is obtained from pilots and defence analysts, it is usually true that facts and rules are neither totally certain nor totally consistent due to the varied experience sets of the pilots. This leads to the reasoning processes used by experts in certain situations as approximate. In this paper, the theory of fuzzy sets is used to help assess uncertain information derived from this approximate reasoning process. Features of Fuzzy Logic that give it a realistic implementation than classical logic are:-1. Fuzzy Logic offers far greater resources for managing complexity and controlling computational cost. More complex the problem involved, the greater the superiority of fuzzy methods. 2. Fuzzy Logic has greater expressive power. It has the capability to capture and deal with meanings of sentences expressed in natural language. This capability allows to deal in mathematical terms with problems that require the use of natural language. 3. Fuzzy Logic has capability to capture human common-sense reasoning, decision making and other aspects of human cognition. 4. Data which are based on graded distinctions among states of relevant variables are usually called fuzzy data. When fuzzy data are processed, their intrinsic uncertainties are processed as well, and the results obtained are more meaningful, than their counterparts obtained by processing the usual crisp data. In any given language the values of a linguistic variable are words, phrases, or sentences. For example, structural damage can be considered as a linguistic variable with values such as "severely damaged," or "moderately damaged." These are meaningful classifications but not clearly defined. With the use of fuzzy sets, however, we can quantify such terminology and apply it in a meaningful way to help solve a complex problem. An evident advantage of the fuzzy set approach is the possibility of representing numeric and linguistic variables in a uniform way and of using a formalized calculus to manipulate these variables [START_REF] Vijayrao | The Design of Air Wargame System Simulation system[END_REF], [START_REF] Mendel | Uncertain Rule-Based Fuzzy Logic Systems-Introduction and New Directions[END_REF].

Identification of Factors for Damage assessment

The metric "comparative damage relative to intended damage" gives a measure of damage caused to the target relative to the damage that was intended to be caused, when the weapon was delivered. It uses various factors as its input variables such as Offset, Weapon Suitability, Weapon Delivery Mode, Target Identification factor and Terrain. Meetings with the pilots and analysts helped in identifying following parameters to be considered while assessing damage.

(1) Offset is the measure of deviation of actual point of impact from desired point of impact. It is calculated using actual altitude, actual vertical flight path angle, actual wind speed and observed altitude, observed vertical flight path angle, observed wind speed by the weapon system trajectory calculation module and the aircraft speed as the input variables. Offset is a measure of Altitude induced error, wind induced error, vertical flight path angle induced error. [START_REF] Vijayrao | The Design of Air Wargame System Simulation system[END_REF], [START_REF] Meitzler | Fuzzy Logic Technique to Determine Search Time and Probability of Detection for Targets of Interest in Background Scenes[END_REF], [START_REF] Celmiħs | Vulnerability assessment of fuzzy targets[END_REF], [START_REF] Meitzler | Advances in Target Acquisition Modeling[END_REF].

(2) Weapon Suitability factor takes into account weapon characteristics, weapon's suitability in the operational conditions and weapon-target matching. Apart from the geo-political and threat environment factors such as weapon/target matching, affordability and maintainability are also considered. The weapon is chosen for attack considers parameters such as attack conditions, significance attached with the target versus cost of weapon, platform for weapon delivery, target-weapon match and collateral damage. Weapon Suitability factor is rated on the scale of 0 to 10 considering following factors:--Operational Environment and Weather Conditions: Operational environment and weather conditions play an important role in deciding the suitability of the weapon for a target. For instance, poor weather and conflict induced environmental conditions such as smoke from bombing etc. may degrade or block the IR, EO or laser targeting sensors required for delivery of guidance ordnance.

LGBs cannot be used in poor weather where target illumination cannot be seen, or where it is not possible to get target designator near the target. However, poor visibility does not affect satellite guided bombs. Only comparatively inaccurately unguided bombs could be delivered in poor weather.

-Cost versus Target Significance: The expenditure incurred in using a weapon against a target should be justified taking into account significance attached with the target. For instance, there are high costs associated with the usage of guided munitions; they must not be neglected in cost-benefit analyses and the deliberation processes that are intrinsic to capital equipment purchases. The comparatively high cost of guided bombs and resulting smaller inventories is also an important factor in weapon suitability factor.

-Aircraft Compatibility: Compatibility of the aircraft for weapon delivery is an important factor. Weapons must be compatible for carriage and release with the combat aircraft types.

-Weapon-Target matching: Target size and target hardness are factors that need to be considered for deciding the suitability of a weapon against a target. For large targets, use of unguided ordnance is appropriate. LGB capable aircrafts could deliver penetrating LGBs. LGBs are particularly useful against hardened, reinforced, buried reinforced targets. The aim of matching weapons to targets is to achieve the desired level of damage for the least number of sorties while minimizing the risk to the weapons delivery platform. Weapons employed against certain targets such as ships must be inherently maneuverable so they can accommodate target velocity after launch.

-Collateral Damage: Although advances in weapon aiming systems have improved the accuracy of unguided air-delivered munitions, collateral damage cannot be discounted. PGMs must be employed against military objectives where there is a possibility of collateral damage to civilians or civilian objects.

(3) Weapon Delivery Mode is another factor considered for calculating damage. It takes into account aircraft for weapon delivery, weapon chosen and target characteristics. A value is assigned on the scale of 0 to 10. Three basic delivery modes considered are Roll, Toss and Dive. Weapons can be either direct attack (freefall, accurate, or precision) or standoff. Direct attack weapons require an aircraft to maneuver to a release point. Stand-off weapons enhance an aircraft's survivability by allowing it to deliver ordnance at increased distance from the enemy's defenses. Planners and crews prefer launch-and-leave weapons to those requiring line of sight with the target. These tend to be much more expensive and fewer in number than other weapons.

(4) Target Identification factor: Rapid and certain target detection and identification are the dominant factors in the success of all air-to-ground attacks. The ability of tactical fighters to penetrate enemy defenses and to acquire and identify ground targets successfully is a keystone of success in a mission. Enemy defenses have necessitated higher speeds and higher altitudes. The employment of standoff weapons demands acquisition and identification of targets at even greater ranges. It has been observed that aerial observers respond to targets in a manner indicating that detection / identification represents a continuum rather than discrete phenomena. At one extreme the response is based on the ability to merely discriminate the existence of a military object among non-military objects (detection). At the other extreme the observer can describe the object in precise detail (identification). Factors considered for computing the Target Identification factor are target size, percent contrast, illumination, terrain, altitude and speed of the aircraft at time of target acquisition.

-Target Size: As target size increases, probability of correct target identification increases. It may vary from small to large tactical targets, including personnel, trucks, and tanks to big targets as bridges, runways and taxi-tracks.

-Contrast: Target/Ground Brightness Contrast is expressed as a percentage.

-Illumination: Detection performance increases as illumination increases. Effects of decreases in illumination occurring after sunset and before sunrise are very important and need to be considered.

-Terrain: Types of terrain have been defined in terms such as number of slope changes per unit area and average slope change. Four different terrain types have been defined--fairly smooth, moderately rough, rough, and very rough. As the roughness of terrain increases, percent terrain view decreases, and decrease in detection performance is observed.

-Altitude: The relationship between altitude and target detection/identification is normally one in which there is assumed to be an optimal altitude; above and below this optimum altitude, detection is reduced. As altitude increases, detection performance decreases. As altitude is increased beyond an optimal point, detection probability falls off rapidly. Data on all these factors are collected from handbooks and experimental field trials and subjective decisions from experts and defense analysts (in questionnaire form) are collected and recorded. They are then represented as decision matrices and decision trees which form the basis to design the membership functions and rules. The rules are then executed in the mission processing module and defuzzified to obtain the damage to target. These results are then compared to the expected output and finetuned before storing in the rule base. A decision to include the new rule or not is provided to the commander. Missions and results of the missions are stored as a case-base for retrieval and reuse of missions plans in new situations. The following fuzzy linguistic variables used in the design of the game rules are as follows:

Terrain Case Mission ID # 001: Consider a large area-target of size of 550 ft to be attacked, where the fuzzy variables target-ground contrast 80%, the terrain, rated 8, is fairly smooth, aircraft altitude is 900 ft, aircraft range is 5000 ft is flying at 100 knots speed. The target identification factor for this target is seen as "good" with value 7.3295. In this mission, on firing the rules for inference, the offset from the desired point of impact is 29m, considered "less"(i.e. fairly accurate targeting); weapon-target match is 6 (average), "good" target identification factor 7.3295, the relative damage caused is 28.9187 which is a "moderate" damage to the target. Case Mission ID # 002: We now consider another mission planned by the commander where a similar target is chosen with the fuzzy variables as shown in Tables 1 and2.

While the offset has reduced to 6m, considered "very less" (i.e. very accurate targeting), choosing a different weapon system and delivery improved the weapontarget match to 9 ("good"), and mode of weapon delivery 9, the target identification factor also improved to 8.033 (considered "excellent"), and the relative damage caused is 88.7410, which is a "substantial" damage to the target. (Tables 1,2). These attributes form the antecedents of the fuzzy rule and the consequent is shown in the last column of the tables. For all the missions that the pilots plan in the wargame exercises, these fuzzy game rules are used to infer the expected damage caused to the target. These missions form a part of a case-base which is used as part of the 'learning' by the system for future instructional use.

Conclusions:

We present a novel approach using a fuzzy rule-based system to design the game rules in a mission planning and evaluation system. The conventional crisp cookie cutter function used to compute the probabilistic damage caused to a target is replaced by a fuzzy cookie-cutter function, which takes into account many physical parameters before assessing the possibilistic damage caused to the target. This methodology of damage assessment computation of targets using fuzzy rule bases gave more realistic results when used in field training and deployment of the system.

  : 1-100 {Fairly Smooth [0 -22]; Moderately Rough [14 -49]; Rough [45-81]; Very Rough [75 -100]}. Target Size (in feet): {Very small: 0 -100; Small: 70 -190; Medium sized: 160 -300; Large: 270 -400; Fairly Large: 360 -500; Extremely Large: 450 -900} Damage: Offset (in meters): {Very Less:[0-23]; Less:[16-36]; Medium: [34-57]; Large: [56-80]; Very Large [78-100]} Weapon Target Match: [0 to 10] {Poor: [0-3.6]; Average: [3.36 -6.669]; Good: [6.73 -14.2]} Target Identification Factor: [0 -10] {Very poorly identified: [0-1.19]; Poorly identified [0.96 -2.43]; Average identification [2.34 -5.61]; Good identification [5.43 -7.55]; Excellent identification [7.35 -10]} Relative Damage (Damage relative to intended damage): [0 -100] {Mild: [0-18]; Moderate: [16-36]; Average: [34-57]; Severe: [56-80]; Fully Damaged: [78-96]} These fuzzy variables to calculate the target identification factor are depicted in the Mamdani FIS as shown in Fig.1 and firing of the rules to compute the relative damage to the target are depicted in Fig. 2. Offset is calculated using Actual altitude, actual vertical flight path angle, actual wind speed and Observed altitude, observed altitude, observed vertical flight path angle, observed wind speed by the weapon system trajectory calculation module and the aircraft speed as the input variables (Table3). Offset is a measure of induced error, wind induced error, and vertical flight path angle induced error.

Fig 1 :

 1 Fig 1: Mamdami Fuzzy logic Inference System for computing Target Identification factor

Fig 2 .

 2 Fig 2. Fuzzy Rules and their firing for the Mamdami FIS for finding Damage caused in relation to the intended damage.Table1: Fuzzy Rules to determine the Target Identification factor

Table 1 :

 1 Fuzzy Rules to determine the Target Identification factor

	MissionID	Target	Target-	Illumination	Terrain	Aircraft	Aircraft	Aircraft	Target
		Size(ft)	Ground	(foot candles)		Altitude(feet)	Range(feet)	Speed	Identify
			Contrast%					(knots)	Factor
	#001	550	80	40	8	900	5000	100	7.3295
	#002	550	80	60	7	750	4000	80	8.0333

Table 2 :

 2 Fuzzy Rules to compute the Relative damage to target

	MissionID	Offset	Target	Weapon-Target	Weapon	Target Identify	Relative damage
		(meters)	Radius (km)	Match	Delivery Mode	Factor
	# 001	29.027	0.0900	6	6	7.3295	28.9187
	# 002	6.0696	90	9	9	8.0333	88.7410

Table 3 :

 3 Fuzzy attributes to determine the offset of the weapon from the intended target

Acknowledgments. The authors thank the Director, ISSA, for his guidance, support, encouragement and permission to present this work. We also thank Air Cmde S P Ojha, College of Air Warfare and Air Mshl. P K Mehra for the useful insights and discussions on air warfare tactics and game rules development; and the scientists of the air wargame team for their enthusiasm and support.


Ioannis Hatzilygeroudis

Konstantinos Kovas
email: kobas@ceid.upatras.gr

A Tool for Automatic Creation of Rule-Based Expert Systems with CFs

Keywords: expert systems, certainty factors, unsupervised learning, MYCIN, genetic algorithm, CLIPS

This paper introduces a tool, namely ACRES (Automatic CReator of Expert Systems), which can automatically produce rulebased expert systems as CLIPS scripts from a dataset containing knowledge about a problem domain in the form of a large number of cases. The rules are created via a simple systematic approach and make use of certainty factors (CFs). CFs of same conclusions can be combined either using the MYCIN method or a generalization of MYCIN's method. This latter method requires calculation of some weights, based on a training dataset, via the use of a genetic algorithm. Creation of an expert system is outlined. Small scale experimental results comparing the above methods with each other and a neural network are finally presented.

Introduction

A lot of datasets are available nowadays, containing known instances from various domains (e.g. UCI Machine Learning Repository). Effort is required to retrieve information from these datasets and use them as a knowledge base for predicting new instances. A tool is therefore needed that can automatically turn these raw data to rules that can classify new data accordingly using a model of uncertainty. Existing efforts at providing such automated tools are mostly commercial and thus targeted to professionals or knowledge engineers (e.g. [START_REF] Awad | Building Knowledge Automation Expert Systems with Exsys Corvid[END_REF]), making the process difficult for intermediate users or domain experts that want to make use of existing knowledge in datasets. The uncertainty models implemented are also not efficient in many cases or lack the ability to adjust to the requirements of each individual problem. The main goal of this work is to introduce a tool (ACRES), which can make use of datasets and other available sources of knowledge, in order to easily create expert systems that use that knowledge to cope with new instances of a problem. A generalized version of the MYCIN certainty factors (CFs) is used as the way to represent uncertainty, allowing optimization of the expert system. In section 2 we present existing tools for the creation of expert systems. In section 3 we present certainty factors as a way to represent uncertainty in expert systems and present the two methods we implemented. The way the automatic generation of expert systems from datasets can be achieved is described in Section 4. In section 5, an example creation of an expert system is outlined. In section 6, some experimental results are presented. Finally, section 7 concludes and discusses future works.

Related Work

One of the most well-known tools for developing rule-based expert systems is CLIPS (C Language Integrated Production Systems) environment from NASA [START_REF] Culbert | [END_REF]. JESS (Java Expert System Shell), its Java counterpart, has also received great interest [START_REF] Hill | Jess in action: rule-based systems in Java[END_REF]. Expert System Creator [START_REF] Pop | Knowledge Management in Expert System Creator[END_REF] is a commercial software tool for the development of expert system based applications, representing domain knowledge using rule sets, decision tables or classification trees. It is mainly targeted to professionals, domain experts or knowledge engineers and less for simple users. EXSYS [START_REF] Awad | Building Knowledge Automation Expert Systems with Exsys Corvid[END_REF], an also commercial Expert System Development Software, is easier to use and provides many different uncertainty models to choose from. Our proposed tool focuses on the automatic generation of expert system knowledge bases from existing datasets with the minimum required effort from the user. At the same time it offers the ability to optimize the output expert system taking into account specific requirements or priorities set by the developers-users.

Expert Systems and Certainty Factors

Combining Conclusions with Certainty Factors

MYCIN [START_REF] Buchanan | Rule-Based Expert Systems, The MYCIN Experiments of the Stanford Heuristic Programming Project[END_REF] was a medical expert system developed in Stanford University in the early 1970s. It was the first one that introduced Certainty Factors (CFs) as a way to represent uncertainty when a conclusion is made by a rule. Although CFs have some problems [START_REF] Hackerman | From certainty factors to belief networks[END_REF], they still remain a simple and useful way of dealing with uncertainty.

CFs are associated with a simple computational model that permits to estimate the confidence in conclusions being drawn. A CF is a number between -1 (definitely false) and +1 (definitely true), which measures the expert's belief (positive number) or disbelief (negative number) to a conclusion. However, usually CFs that are positive numbers between 0 (definitely false) and 1 (definitely true) are resulted, due to the nature of most problems. In these cases, any CF less than 0.5 indicates disbelief, whereas any CF equal to or greater than 0.5 indicates belief in a conclusion. Given that CFs are positive, when we have the following rules with the same conclusion and CF1, CF2 respectively and they are fired, the combined certainty CF for conclusion, according to MYCIN theory, is given by the formula:

CF = CF 1 + CF 2 (1-CF 1 ) = CF 1 + CF 2 -CF 1 * CF 2
(1)

Generalizing Certainty Factor Computation

The above formula didn"t give satisfactory results in many cases. So in the expert system PASS [START_REF] Hatzilygeroudis | PASS: an Expert System with Certainty Factors for Predicting Student Success[END_REF], a generalized version of (1) was introduced:

CF = w 1 * CF 1 + w 2 * CF 2 + w * CF 1 CF 2 (2)
where w1, w2 and w are numeric weights that should satisfy the following equation:

w 1 + w 2 + w = 1 (3)
to assure that 0 ≤ CF ≤ 1.

To use formula (2), however, the weights w 1 , w 2 , w should be first determined. In PASS, statistical data about the problem was used, as a training data set to determine the weights by hand. The tool presented in this paper computes the above weights automatically, utilizing a genetic algorithm.

Automatic CReation of Expert Systems (ACRES)

Rule and CF Generation

One of the main functions of the tool is to convert raw data in a dataset to a set of rules. We consider that each instance of the data set contains discrete values corresponding to attributes related to a problem domain. One of the attributes represents the class (or output) attribute. We consider binary class attributes, i.e. datasets referring to two classes (A and B). So, class attribute takes one of two values, e.g. yes-no or true-false. We distinguish between positive instances (those belonging to the class "yes" or "true") and negative instances (the rest ones). Rule and CF generation is then achieved by the following process:

1. Cluster instances in groups, so that each group contains instances that have identical values for all non-class attributes. 2. From each such group produce one rule with as conditions the attribute-value pairs of the instances and as conclusion the class attribute-value pair (where the value is the one of the positive instances). 3. Associate with each rule a CF defined as CF = n p /N where n p is the number of the positive instances in the group and N the number of all instances. That is, a CF is defined as the frequency of the positive instances in the group.

Expert System Creation

ACRES can produce one CLIPS-based expert system (ES) from a given data set. The process is as follows:

1. Specify the attributes and the class attribute of the data set. 2. Divide attributes in two possibly (typically) overlapping groups. 3. Extract from the dataset two subsets, corresponding to the two attribute groups. Each subset includes instances having values for the attributes of the corresponding group and the class attribute. 4. Produce rules with CFs separately from the two subsets using the process in Section 4.1. 5. Produce the rule implementing computation of the combined CF, based on formula (2) above. 6. Produce the basic expert system 7. Determine the weights w 1 , w 2 and w of formula [START_REF] Buchanan | Rule-Based Expert Systems, The MYCIN Experiments of the Stanford Heuristic Programming Project[END_REF]. The rationale behind the division of the attributes and the dataset in two groups/subsets is the need to have two stages of reasoning or two alternate ways of reasoning for the same conclusion, from which the need for CFs combination comes.

In step 7, if we want to produce a MYCIN-like expert system, the system sets w 1 = w 2 =1 and w = -1. If not, the genetic algorithm is used.

Genetic Algorithm

We use a simple genetic algorithm to determine the best values for weights w 1 , w 2 and w, since it is an optimization problem. An initial population of possible solutions, called atoms (or chromosomes or genomes), is created. Then the atoms of the population are evaluated via a fitness function and the best of them are selected to pass to the next generation after having been processed through two processes, i.e. crossover and mutation, which resemble corresponding genetic processes. The algorithm stops when any of the set termination criteria is met.

In such an algorithm, there are two crucial design decisions: the representation scheme of an atom and the fitness function. In our case, an atom is a representation of the values of weights w 1 and w 2 , since w can be calculated as w = 1-(w 1 + w 2 ).

The fitness function should evaluate how well the represented weight values in an atom affect the effectiveness of the expert system under creation. Usually, three metrics are used for evaluating classification-oriented expert systems: accuracy (abbr. acc), sensitivity (abbr. sen) and specificity (abbr. spec) [START_REF] Hall | The WEKA Data Mining Software: An Update[END_REF][START_REF] Hill | Jess in action: rule-based systems in Java[END_REF]. So, the fitness function should be based on them. Although accuracy is the most important, sensitivity and specificity should be taken into account too. Also, sometimes a system is required to give more emphasis to one of them. In our GA implementation, the fitness function is defined as

fitness = w ACC * acc + w SEN * sen + w SPE * spec + w BAL * bal ( 3
)
where w ACC , w SEN , w SPE and w BAL are user-defined factors and "bal", which is an abbreviation for "balance", is defined by the following expression:

bal = 1-|sen-spec| (4)
which represents how balanced sensitivity and specificity are.

With the above definition we can define a variety of fitness functions, depending on which metric and in what degree we consider as more significant for the under creation expert system.

To compute fitness for each atom of the population, we need to evaluate the expert system under construction. So, the CLIPS engine is embedded in our system and is called to run the constructed expert system for all instances of the dataset. Based on the results we compute the values of the three metrics and then the fitness for each instance.

Systems Comparison

As is stated above, ACRES can produce two types of expert systems with CFs, one of MYCIN type and one of WEIGHTED type. The second type uses the generalized way of CF combination. This is related to the expert system creation mode of the tool.

However, apart from that, ACRES give the possibility to compare the two types of systems. So, in its method comparison mode, the system creates two similar systems from the same dataset, one of MYCIN and one of WEIGTHED type, and compares them. Comparison process is as follows:

1. Split the dataset in two sets, a training and a test set.

2. Produce the basic expert system using the training set as the data set.

3. Produce the two expert systems (the MYCIN-type and the WEIGHTED-type). 4. Evaluate the two systems (i.e. compute acc, sen and spec) using the test set. 5. Repeat steps 1-4 k times (k is defined by the user). Repetition of steps 1-4 actually implements a k-fold validation process.

Implementation Issues

Both modes were implemented as console applications with C++, integrating the CLIPS library to load the expert systems created and the GALIB library for the genetic algorithm. The system uses a population size = 15, p c = 0.7, p m = 0.05 and terminates after 50 generations. This last parameter was chosen after having made a number of experiments. The graphical user interface was developed with Visual C++.

Example Use of the Tool

This section presents an example use of the presented tool, using a dataset from the UCI repository [START_REF] Hackerman | From certainty factors to belief networks[END_REF]. The breast cancer dataset was chosen, containing 286 instances of patient instances.

There are 10 variables in the dataset. The user should prepare two files. The one is the dataset, as provided in the UCI repository. In our case, it is the "breast.data" dataset file. The second file is the variables file ("VariablesNames.txt"), which contains the names of the attributes of the problem related to the dataset. The first variable is the class variable. In our case, it has two classes: "no-recurrence-events" and "recurrence-events".

Figure 1. ACRES Graphical User Interface

In Fig. 1 the graphical user interface is shown, in the expert system creation mode. The user can browse for the above files. After loading them, he/she can choose the output/class variable. Next the user specifies two variable groups for which rules will be created. The way this is done is something that requires some knowledge about the domain (an expert"s advice would help here). The groups can have common variables. Then, he/she can choose to create a MYCIN type system and/or a WEIGHTED type one. If the latter type is chosen, the GA parameters for the fitness function should be specified.

In the method comparison mode we can evaluate the two methods for combining conclusions. The user can specify the train/test datasets ratio and the cross validation folds. We"ve used 1/3 for the test/train ratio and 3-fold cross validation. After pressing the "Compare Methods" button the results of the evaluations of the two systems are depicted. Based on the results one can decide which of the methods is more appropriate for the problem and switch to the "expert system creation" mode. In our example, the WEIGHTED version of the expert systems does better.

The expert system is created as a CLIPS program file. The program consists of four template definitions. The first template, named "data", has as slots the variables of the dataset. The next two templates, named "result1" and "result2", respectively keep the results (classes and corresponding CFs) of the two reasoning stages for the two classes. The fourth template, named "final" includes the final results from the combination of two different stage rules. The program also includes two groups of rules, one for each group/stage of variables/reasoning. Finally, it contains a rule that combines the results of two different stage rules. In Table 1, two rules, one from the first group and one from the second group of rules are presented, whereas Table 2 shows the final combination rule for the WEIGHTED version of the expert system (where w 1 = 2.72175, w 2 = 0.666972, w= -2.38872). (defrule ResultCombination (result1 (no-recurrence-events ?r1_0) (recurrence-events ?r1_1)) (result2 (no-recurrence-events ?r2_0) (recurrence-events ?r2_1)) ?y1<-(result1 (no-recurrence-events ?r1_0) (recurrence-events ?r1_1)) ?y2<-(result2 (no-recurrence-events ?r2_0) (recurrence-events ?r2_1)) =>(bind ?r_0 (+ (* ?r1_0 2.72175) (* ?r2_0 0.666972) (* ?r1_0 ?r2_0 -2.38872) ) ) (bind ?r_1 (+ (* ?r1_1 2.72175) (* ?r2_1 0.666972) (* ?r1_1 ?r2_1 -2.38872) ) ) (assert (final (no-recurrence-events ?r_0) (recurrence-events ?r_1)) ) retract ?y1 retract ?y2)

The CLIPS file can then be loaded in the CLIPS Expert System shell and make predictions about new instances of the problem.

Experimental Results

We used ACRES to produce two expert systems, one of MYCIN-type and the other of WEIGHTED-type) for predicting the success (or failure) of a technical high school student to the National exams in Greece. We used a real dataset consisting of 373 instances (student records). There were five attributes plus the class attribute. We also used WEKA [START_REF] Hall | The WEKA Data Mining Software: An Update[END_REF] to produce a neural network for the same target. We used the same training and test sets for all systems and a 5-fold cross validation.

From Table 3, where results are presented, it is concluded that our ACRES weighted method of creating expert systems is comparable and in cases better than neural networks method. Additionally, construction of such systems in ACRES is much easier and controllable.

Conclusions and future work

We present a tool that can be used to easily create an expert system from existing knowledge stored as instances in a dataset. The tool introduces a method for creating rule-based expert systems with CFs in a CLIPS-based format. The method results in a two stages rule-based reasoning with CFs whose results are combined using a generalization of MYCIN"s policy. The tool also allows for comparison of the two types of systems, the MYCIN-based one and the WEIGHTED one, which is based on the generalized formula for CF propagation. Experimental results are promising.

There are however a number of possible improvements that are opportunities for further work. For example, the way the variable groups are selected is currently something completely assigned to the user. A way to provide him with a tool that can detect dependency formations between the dataset variables would be helpful. Also, regarding the input dataset, the tool requires that its variables have discrete values. If they don"t, the user must take care of discretizing them. So, a facility helping towards this target is desirable. A more advanced solution to this would be the use of fuzzy logic combined with CFs (in a FuzzyCLIPS style) in the produced rules.

Table 1 .

 1 Representative rules from the two rule groups

	(defrule group1_1	(defrule group2_10
	(data (age 40-49)	(data (inv-nodes 3-5)
	(menopause premeno)	(node-caps yes)
	(tumor-size 20-24))	(deg-malig 2))
	=>	=>
	(assert (result1	(assert (result2
	(no-recurrence-events	(no-recurrence-events
	0.823529)	0.583333)
	(recurrence-events	(recurrence-events
	0.176471))))	0.416667))))

Table 2 .

 2 Rule for combination of results

Table 3 .

 3 Comparison of methods

	METRIC	MYCIN	WEIGHTED	NN
	acc	0.7334	0.8152	0.803
	sen	0.7446	0.5639	0.630
	spe	0.7291	0.9095	0.869


Tijs VAN DEN BROEK Silja Eckartz

Merel Ooms

Open Data Innovation Capabilities: Towards a Framework of How to Innovate with Open Data

Keywords: Open data, Re-Use, Public Sector Innovation, Value of Data and Apps, Capabilities

Innovation based on open data lags behind the high expectations of policy makers. Hence, open data researchers have investigated the barriers of open data publication and adoption. This paper contributes to this literature by taking a capabilities perspective on how successful open data re-users create value out of the available data sources. First, a framework of IT, organization and skills capabilities required to innovate with data is derived from literature. Second, a case study including a survey and interview with managers from 12 frontrunners in the Netherlands was conducted. The analysis reveals that skills are valued the highest closely followed by organizational capabilities. Setting up a multidisciplinary team with motivated employees and giving this team the mandate to experiment with data, is essential when innovating with open data. Theoretically, this study contributes to open data research by offering a new capabilities perspective on the organizational level. Our results highlight the importance of entrepreneurship theories to explain value creation with open data. Practically, our study suggests that digital skills and start-ups are important to the open government data policies.

Introduction

An increasing number of government agencies release open data to spur economic growth through the development of new digital services or increasing organizational productivity. McKinsey [START_REF] Manyika | Open data: Unlocking innovation and performance with liquid information[END_REF] estimates that open data may provide 900 billion US dollar additional annual economic growth in Europe compared to an economy without open data. To unlock this potential, more and more data sources, ranging from transport to educational data, are available in open data portals throughout Europe. However, the question remains how to further exploit the economic potential of open data. Currently, innovation based on open data sets seems to lag behind the high expectations of policy makers. Hence, previous research took stock of barriers to the publication and use of open data. This paper contributes by taking a capabilities perspective to study how successful open data re-users create both economic as well as social value out of the available data sources. What can we learn from these frontrunners? Hence, the central research question is: "What capabilities are most important to create value out of open data?"

The gap between open data provision and usage is reflected in IS research. To date, most attention has been paid to barriers at the side of open data providers, which are predominantly government agencies [START_REF] Huijboom | Open data: an international comparison of strategies[END_REF][START_REF] Van Veenstra | Opening moves-drivers, enablers and barriers of open data in a semi-public organization[END_REF][START_REF] Zhang | Exploring stakeholders' expectations of the benefits and barriers of e-government knowledge sharing[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF]. Recently, scholars have started to examine the impediments that open data re-users experience [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF]. However, most of these studies focus on the technical barriers related to opening, finding and using data that hinder usage [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF][START_REF] Iodc | Enabling the Data Revolution: An International Open Data Roadmap[END_REF]. Several scholars urge for more research on the non-technical barriers, such as those related to business cases, management support and organizational culture, that open data re-users experience [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Lavalle | Big data, analytics and the path from insights to value[END_REF]. Only a few studies examine the capabilities that open data re-users need to create economic value from data [START_REF] Jetzek | Data-driven innovation through open government data[END_REF][START_REF] Venkatraman | Theorizing Digital Business Innovation: Platforms and Capabilities in Ecosystems[END_REF]. Jetzek et al. [START_REF] Jetzek | Data-driven innovation through open government data[END_REF] describe capabilities as "the collective ability of individuals and organizations to use and re-use open data" and focus on access to open data and data literacy. Complementing Jetzek et al. [START_REF] Jetzek | Data-driven innovation through open government data[END_REF], we aim to study open data innovation capabilities related to data re-use focused on the organizational level.

Our paper contributes to open data research in two ways. First, we synthesize information systems (IS) and innovation management (IM) literature on capabilities for data-driven innovation to develop a conceptual framework. To complement current open data research, this framework includes IT, organizational and people-related capabilities. Second, we present the results of the study of 12 Dutch frontrunners in innovation based on open data to test the conceptual framework. The study includes a questionnaire and semi-structured interviews with the CIOs and/or technical managers of the 12 organizations.

The practical implications of this study are twofold. First, we formulate the capabilities required for organizations that want to start to innovate with open data, e.g. by developing new services or improving their current practices. Firms may use the insights to develop an open data innovation plan that includes technical and nontechnical measures. Second, policy makers may use the insights of this paper to develop policy instruments that stimulate data-driven innovation. For example, our research may fuel policy debate about the digital skills that organizations need.

An Open Data Innovation Capabilities Framework

A literature review identified IS and IM journal articles and conference papers that study capabilities needed for data-driven innovation. Keywords for our search included combinations of the words e-Skills, capabilities, innovation, "data-driven", "digital skills", "data re-users" and barriers. We found 19 relevant studies that examined capabilities for data-driven innovation. The studies are mentioned in the tables below.

We clustered the results of our literature review around three main types of capabilities: IT related capabilities, organizational capabilities and skills-. Each capability type was identified in literature as influencing the data innovation capacity of an organization. Within each concept we distinguish a number of categories and sub categories, presented in the sections below.

In contrast to previous research [START_REF] Venkatraman | Theorizing Digital Business Innovation: Platforms and Capabilities in Ecosystems[END_REF], the selected IS and IM studies point at the importance of non-technical capabilities to innovate with data. This reflects the general notion in innovation management that three quarter of innovations' success is related to non-technical factors [START_REF] Volberda | Concurrentie en Innovatie Monitor[END_REF]. Furthermore, many capabilities are related to small or medium-sized enterprises and not to large enterprises, for example entrepreneurship and agility.

IT Capabilities

With respect to IT capabilities we found three categories that influence the data innovation potential of an organization: 1) Infrastructure and enabling technologies that include all hardware and software that is needed to collect, store, analyze and visualize open data, 2) An IT strategy that includes planning, data management and governance, and measures to secure data, and 3) Interoperability that includes capabilities on working with standards and ability to integrate IS. Table 1 provides an overview of the IT capabilities, how they are defined in literature and a list of studies that mention them.

Organizational Capabilities

With respect to organizational capabilities we found four categories that influence the data innovation potential of an organization: 1) Strategic capabilities that includes top management support for data-driven innovation and the ability to change decisions and policies based on open data, 2) Tactical capabilities that include the mixing of disciplines over organizational silos, allocation of resources, and ability to adapt organizational processes based on data, 3) Operational capabilities that include entrepreneurship and R&D, and 4) cultural capabilities that include a culture focused on innovation and agility. Table 2 provides an overview of the organizational capabilities, how they are defined in literature and a list of relevant IS and IM studies.

Skills

With respect to skills we found two categories that influence the data innovation potential of an organization: 1) Hard skills, and 2) Soft skills. Where hard skills can be defined as more technical skills such as programming or data analytics skills and soft skills as more non-technical such as interdisciplinary cooperation and entrepreneurship. Table 3 provides an overview of the capabilities, how they are defined in literature, and a list of relevant IS studies.

Table 3 Overview of skills identified

Category

Capabilities Definition Authors

Hard skills e-literacy Computer science skills, utilization, Architecture skills, e-user skills [START_REF] Cosic | A business analytics capability framework[END_REF]20,[START_REF] Kabir | Big Data, Tacit Knowledge and Organizational Competitiveness[END_REF][START_REF] Mitrovic | Building Open Data Capacity through e-Skills Acquisition[END_REF][START_REF] Davenport | Data scientist[END_REF] Data literacy Interoperability, semantic webs, linked data skills [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF][START_REF] Hemerly | Public policy considerations for data-driven innovation[END_REF][START_REF] Kabir | Big Data, Tacit Knowledge and Organizational Competitiveness[END_REF][START_REF] Mitrovic | Building Open Data Capacity through e-Skills Acquisition[END_REF] Programming skills Different programming languages, hacking machine learning, NLP, Mapreduce / [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF][START_REF] Hemerly | Public policy considerations for data-driven innovation[END_REF][START_REF] Kabir | Big Data, Tacit Knowledge and Organizational Competitiveness[END_REF][START_REF] Davenport | Data scientist[END_REF] Data analytics Knowledge of data models, cleaning, integration and analysis [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF][START_REF] Hemerly | Public policy considerations for data-driven innovation[END_REF][START_REF] Kabir | Big Data, Tacit Knowledge and Organizational Competitiveness[END_REF][START_REF] Mitrovic | Building Open Data Capacity through e-Skills Acquisition[END_REF] Research skills Research and analysis skills / Statistics [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF][START_REF] Jetzek | Data-driven innovation through open government data[END_REF][START_REF] Hemerly | Public policy considerations for data-driven innovation[END_REF][START_REF] Sharma | Transforming decision-making processes: a research agenda for understanding the impact of business analytics on organisations[END_REF][START_REF] Davenport | Data scientist[END_REF][START_REF] Maruyama | Developing Data Analytics Skills in Japan: Status and Challenge[END_REF] Data output skills Communication skills: Story telling with data, data journalism and visualization [START_REF] Hemerly | Public policy considerations for data-driven innovation[END_REF][START_REF] Davenport | Data scientist[END_REF] Soft skills

Management skills e-leadership skills, decision quality [START_REF] Sharma | Transforming decision-making processes: a research agenda for understanding the impact of business analytics on organisations[END_REF][START_REF] Mitrovic | Building Open Data Capacity through e-Skills Acquisition[END_REF] Customer and service focus Service provisioning and receiving, customer focus.

[26]

Interdisciplinary cooperation

People integration, knowledge management, Inter-disciplinary domain knowledge [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF][START_REF] Cosic | A business analytics capability framework[END_REF][START_REF] Jia | The Conceptualization of Data-driven Decision Making Capability[END_REF] Entrepreneurship Proactivity, e-Business, competitive actions [START_REF] Cosic | A business analytics capability framework[END_REF][START_REF] Sharma | Transforming decision-making processes: a research agenda for understanding the impact of business analytics on organisations[END_REF][START_REF] Knabke | Exploring the Future Shape of Business Intelligence: Mapping Dynamic Capabilities of Information Systems to Business Intelligence Agility[END_REF]20,[START_REF] Mitrovic | Building Open Data Capacity through e-Skills Acquisition[END_REF]

Methods

To We used a two-step approach to measure the capabilities in our conceptual framework. First, we operationalized the capabilities in the framework in a questionnaire that included seven questions on the organization, their application of open data and which capabilities they find most important. We tested and improved the questions on comprehensiveness. The questions in the survey asked respondents to rate the importance of each capability to the success of creating value out of open data on a five-point Likert scale (ranging from very unimportant to very important). In addition to the closed questions, we allowed respondents to add their own capabilities and explain why they are important. After we received the answers on the questionnaire, we contacted each respondent within one week after filling in the questionnaire for a semistructured interview. In this interview, we asked respondents to explain their answers: why are certain capabilities important or not important at all in their view? Interviews were conducted by phone and varied in length from 15 to 45 minutes. The questionnaire (in Dutch) and interview protocol are available on request.

Table 4 below provides an overview of the frontrunners that participated in our research. Our respondents focus less on the optimization of internal processes and decision support with open data. This might be related to the relatively small average size of the organizations (mainly SMEs).

Results

In this section we describe how respondents rated and explained the importance of IT, organizational and people-related capabilities for the use of open data. The focus in this paper is more on the underlying reasons that organizations have for finding certain aspects important in relation to data innovation than to have representative scores. This is reflected in the modest amount of respondents. Therefore, the figure does not present representative numbers but a reflection of the average of our respondent group.

Information Technology Capabilities

The factors that were found to be most important are data governance (4.40, on a scale from 1 to 5), followed by data management (4.25) and interoperability (4.22). The factors that received the lowest scores were Hardware and Information Systems (3.42) and having an IS strategy and planning in place (3.42).

Looking at a higher level, we can conclude that IT capabilities related to the handling of open data are perceived as most important. First, data governance, which describes the processes and policies that ensures that important data assets are formally managed, is essential for data re-users as it ensures that data can be trusted and that data owners are accountable for risks, for example a low data quality. Specifically, the ability to judge data quality was mentioned by most respondents as being essential to evaluate if data can actually be used. Knowing who the data owner is, how the maintenance of the data is deployed and who is responsible for the quality of the data are success factors when innovating with open data. Another factor related to data governance is to define rules on how to deal with new technological developments related to the data, e.g. cloud storage. Second, respondents rated data management, which comprises all the disciplines related to managing data as a valuable resource, as a very important IT capability when innovating with open data. It is important to specify how data are stored, manipulated and processed, in particular when open data

IT capabilities

Organizational capabilities People capabilities are combined with proprietary data for new services. Third, innovation often means that data re-users need to be able to access different datasets and link them in new ways. Hence, interoperability, semantic standards and system integration are important capabilities, or as one of the respondents indicated: "without standards there is no integration". The ability to handle different open data standards (both open source and proprietary) is important to ensure that data can actually be reused, e.g. data is readable, actual and relevant. The availability of standards makes the reuse of open data faster and more efficient. Data re-users need to be familiar with and their information systems need to be compatible with the latest data standards of the data providers. These standards are domain specific and require domain knowledge. Data analytics technologies, such as software and tools were found to be important but as they are widely (commercially) available, this factor is not considered a challenge anymore. The same holds for hardware and software which are seen as a prerequisite for data innovation but are also considered as given and thus not a challenge anymore to invest in. Of all IT capabilities, having the right hardware and software is deemed the least important. However, just having the right hardware and software does not guarantee that they are used to their full potential. Hardware and software becomes an important capability when privacy, security and ethical questions of the use open data start to arise. With respect to data security it is important that the infrastructure used ensures that the data is stored securely and that data access is in line with access policies. Last, an IT strategy and planning seems to be less relevant to our respondents, which might be biased due to the relatively high amount of small and innovative firms in our sample.

Organizational Capabilities

The organizational capabilities that were found to be most important by our respondents are entrepreneurial orientation (4.55, on a scale from 1 to 5) and an innovative organizational culture (4.55), followed by the organizational capability to develop new ideas (4.45). The organizational capability that was perceived by far as least important for data-driven innovation is the organization's capability to efficiently allocate resources (3.36).

We divided the organizational capabilities into strategic, tactical, operational and cultural capabilities. Based on the questionnaire and interviews we conclude that the operational and cultural organizational capabilities are considered to be most important when innovating with open data.

Two third of our respondents indicated that an entrepreneurial orientation is very important and that, similar to other types of innovation, it comes down to entrepreneurship. This result can be influenced by the fact that the majority of our respondents are smaller, innovative and often more entrepreneurial organizations. To create value out of open data, organizations need teams that are capable of making links between different subjects, develop a network in and outside the organization, recognize opportunities that open data sources offer and show willingness to exploit these ideas. While this capability will come back later as an important skill, it also requires an organizational culture which supports innovation and entrepreneurial thinking by e.g. handling short lines between different management levels and working in small teams. Consequently, respondents indicated that organizations, in order to innovate, need to provide an environment where experimenting with data is encouraged, which is a creative process that may take a while before it creates value. People in organizations with such a mindset think in terms of possibilities and not in terms of barriers. They are not afraid of potential hurdles and challenges but just get started. These organizations are often data driven, have a long term vision and have a culture where innovative teams can play around with data and failure is acceptable. It is best to start an open data innovation project with a small team that has a strong mission and believes in open data innovation. In time this team can take small steps and expand with more and more people that share the same mindset. Moreover, an innovative culture implies that the costs of the innovation e.g. data applications are positioned as costs for the business, not as costs for the IT department. Thus, making the business responsible for data innovation is essential for its success. As the context and challenges around the data might constantly change it is essential to have a learning, agile organization that can quickly adapt to changes in the provision of data.

Multidisciplinary teams are crucial in the open data innovation process, as organizations need people with both technical knowledge and people that understand the domain context of the data and the needs of potential customers. Teams need to understand the bigger picture of the data and be able to talk across boundaries of systems and organizations. Multidisciplinarity helps teams to understand and translate technical complex concepts in something that is visually appealing.

Our respondents indicated that central leadership in itself is less important to innovate, but having mandate and budget from top management is very relevant for the success of an open data innovation. Although innovation initiatives and ideas are most likely to develop on business level it is very important to have a data driven vision at board level to move from an experimental phase to a next phase where innovations are further deployed and implemented. Our interviewees further mentioned that organizations should use successful examples as inspiration to stimulate the re-use of open data in their own organization. Central coordination of resources and activities as well as asset orchestration was found to be the least important in supporting the innovation. Most of the small innovative and entrepreneurial organizations in our sample did not recognize this capability as being important. However, coordinated actions might be valuable for organizations to match the supply and demand of data and stimulate the dialog between data owners and re-users. The management of an organization might want to play a catalyst role in this.

We find that the larger organizations in our sample find it important that the overall organization is ready and willing to support processes and decision-making based on open data. If an organization really wants to innovate with data it should let go of its classical decision making process and use the results of its data innovation to form its own decisions and policies.

Relationship management is a capability that was not initially considered based on our literature analysis and questionnaire, but based on the interviews the ability to engage with external stakeholders (e.g. end-users) is found to be important to be able to innovate. Often, the availability of open data sets is a first step, and close collaboration between the data provider and user is necessary to take the next steps and create value. As one of the respondents said: "Do not think in organizational silos but look across organizational boundaries when innovating with data."

Skills

The factors that got the highest scores and are found to be the most important skills are data analytics skills (4.67 on a scale from 1 to 5) and entrepreneurship (4.64) followed by general e-literacy (4.45) and the capability of people to cooperate between disciplines (4.45). The factor that received the lowest score and was found to be less but still important when innovating with open data are data visualization and reporting skills (3.83). This shows that a successful organization or team consists of people with data analytics skills in combination with entrepreneurship skills and who are able to combine knowledge from several disciplines.

Regarding technical or hard skills, our respondents foremost stressed that it is important to "employ people who can detect patterns in data, who are creative and have a high quantitative skills". Even though respondents valued a team with a diverse set of skills, most team members should have a basic understanding of data (e.g. experience with working with large databases) and good analytical skills to actually make technical sense out of the data. One respondent noted that "it is more important to have mathematicians than having only programmers. The latter you can simply hire". General IT skills, such as programming, are still important when you innovate with open data: "if you do not have your IT Skills in order, then you are not part of the game anymore". However, IT skills just qualify, but a deep understanding and creativity with data gives a competitive advantage. Consequently, open data re-users need employees that have R&D skills to experiment (or 'play' as respondents qualified this skills) and test how different data sources can provide value or insights about the problems they want to address. More specifically, employees need to think in large distributed systems and make complex data combinations (from both open and closed data source).

Regarding non-technical or soft skills, having employees who are team players is key. Respondents valued employers that are open to the ideas of others and who are willing to learn. These quick learners need to be good networkers with a mindset to create new ecosystem, that are a able set up new collaborations with new parties and defend and their business decisions taken based on data. Additionally, it is important to have people with the capacity to think in terms of customer needs. Business models based on open data are often still a challenge. For example, one of the respondents see applications based on open data as a showcase to get to know new customers and sell additional data services. Hence, it is important to start early in the process with the identification of customer needs and business potential.

Discussion

Comparing the three types of capabilities we see that all were found to be important but accents differ. On average skills are valued the highest (4.27) closely followed by organizational capabilities (4.19). IT capabilities are with a score of 3.86 rated the lowest by our respondents. This may indicate that investing in technology and other IT related capabilities that allow you to process and analyze data may not be enough when you want to create value from the re-use of open data. It is seems even more important to focus on skills and organizational capabilities. For example, a lack of employees with the right skills is the biggest barrier to open data innovation.

Although we address the three capabilities in isolation there are many interdependencies between the capabilities. Future quantitative studies with a higher number of respondents may conduct factor analysis to cluster capabilities into a more coherent set of open data innovation capabilities. In addition, some capabilities are present on multiple levels. For example, entrepreneurial orientation (organizational) and an entrepreneurial mindset (people) are very similar and probably overlap.

Respondents stress the importance of an emergent innovation strategy: you need to keep your goal in mind, but how and what IT resources are needed may change rapidly. Organizations need to know where they are going, however this does not need to be planned completely in an IT strategy. This finding may contradict with IS Strategy literature that emphasizes the importance of a well-planned IT strategy that are more important to large organizations. SMEs employ more emergent and flexible strategies. Furthermore, we find that frontrunners focus strongly on open innovation: networks are important in the innovation process to make optimal use of partners' competences and be able to complement each other in the open data innovation process.

While open data is in some aspects different from internal, proprietary organizational data some experts expect that the barriers related open data are largely the same. However, open data may be less sensitive to privacy risks than proprietary data. The value of open data may be the highest when an open data sets are combined with proprietary datasets. Such a combination can bring new insights which would not have been possible without open data.

The small number of organizations studied and the focus on the Netherlands is a limitation of our research which does not allow us to generalize the results to a larger amount of organizations in different national contexts. The surveys and interviews, however, provide interesting insights for future research on the re-use of open data. Future research may expand the number of cases in different countries over a longer period of time . A longitudinal study is required to study how dynamic capabilities may change over time, for example along the maturity of the data innovator.

Conclusion

This paper aims to study which capabilities are required to innovate successfully with open data. We present an open data innovation capabilities framework based on a literature review on capabilities for data-driven innovation. Our analysis of 12 Dutch open data front runners reveals that for innovation with data to happen it is not enough to just make more and more open data available by government agencies and focus on IT capabilities at the side of data re-users.

Theoretically, we contribute by offering a capabilities perspective on open data innovation that complements earlier studies on the barriers of data availability and reuse [START_REF] Huijboom | Open data: an international comparison of strategies[END_REF][START_REF] Van Veenstra | Opening moves-drivers, enablers and barriers of open data in a semi-public organization[END_REF][START_REF] Zhang | Exploring stakeholders' expectations of the benefits and barriers of e-government knowledge sharing[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF]. As data-driven innovation seems to require entrepreneurship, open data research may benefit from theories in entrepreneurship literature, such as dynamics capabilities or causation/effectuation, to better explain the value creation process [START_REF] Bharadwaj | Digital business strategy: toward a next generation of insights[END_REF] Practically, digital skills and digital entrepreneurship policies are as important as persuading governments to open-up data. Policy makers may need to set up stimulating programs that are aimed at educating more employees and entrepreneurs with the right set of skills. Organizations aiming to create more value out of open data may follow the following lessons from frontrunners: 1) Set up a multi-disciplinary team with motivated and creative employees, give this team the mandate to experiment with data, and let them formulate an emergent data strategy and get top management support to create a stable innovation environment. 2) Take an entrepreneurial approach: Recognize and exploit opportunities and strive for an open and experimental culture. When successful, the start-up team may motivate and engage more teams within the organization. 3) Think outside organizational boundaries and silos. Set up new inter-organizational networks with complementing skills and experience. The social interaction between

Figure 1

 1 Figure 1 Overview of how capabilities were rated on a scale from 1 to 5

Table 1 Overview of IT capabilities identified

 1

	Category	Capabilities	Definition	Studies
		Hardware & Software	General IT infrastructure (incl. tools value from data and applications) needed to create	[5, 9, 10, 12, 13]
			Technologies (software and tools) for
	Infrastructure &	Data analytics	data search, discoverability and	[5, 12, 14-16]
	Enabling Technology		mining
			Technologies for showing insights
		Data output/ visualization	from data analytics, including visualization and reporting	[10, 14, 15]
			technologies
		Planning	Vision and IS portfolio management , knowledge management, architecture	[9, 13, 17-19]
	IT Strategy	Data Management	Data lifecycle management, technical data manipulation	[12, 14, 18]
		Data Governance	Processes and policies including data quality and usability	[7, 12, 15]
		Security	System and data security processes and mechanisms	[20]
	Interoperability	Standards System Integration Systems integration, IT integration, Interoperability/ Standards	[5, 15] [14, 15]

Table 2 Overview of organizational capabilities identified Category Capabilities Definition Studies

 2

		Leadership	Top management support	[14, 20]
	Strategic	Governance	Data-driven policies and decision-making	[13, 14, 16, 17]
	Tactical	Multi-disciplinarity	Integration capability, multi-disciplinary teams	[12, 18, 21, 22]

  answer our research question, we studied 12 leading open data re-users in the Netherlands. We decided to sample cases from the Netherlands as the research team had access to Dutch cases and the Netherlands has a lively open data community with governments opening up more and more datasets. However, despite the Dutch central government wishes the Netherlands to be an open data frontrunner in Europe, policy evaluations indicate that this still requires significant effort.1 The open data frontrunners in this study were selected based on their presence as showcase in national and European open data research (e.g. the former ePractice portal and the European Commission's join up community) or because they have won national and European app awards. Furthermore, we selected cases that vary in size and sector to avoid selection bias. The selection process resulted in a longlist of 17 Dutch frontrunners of which 12 respondents finally participated in our research. Frontrunner only participated in the survey. The response rate of 71% can be qualified as very high, which indicates that respondents were highly involved with the topic and therefore more inclined to participate. Our research targeted the frontrunners' technical managers, innovation managers, CIOs or CTOs. Usually, only one respondent per case was contacted.

Table 4 Description of cases Case Description and goal of open data usage

 4 An analysis of the cases shows that two third of our respondents have more than 3 years' experience with open data. About half of our cases are organizations with less than 10 employees. Only 2 cases are large organizations. Most organizations (91.7%) use open data to develop new products or services or improve existing products (58.3%). Half of the organizations aim to develop societal value with open data. Examples of societal value created by the organizations are insight in quality of schools or the plans of the local government in your neighborhood.

					Respondent	Size (nr of	Type of	Years of
					Function	employees)	organization	experience
								with open
								data usage
	A	Development		and	Chief	11-50	App	More than 5
		improvement	of	new	concept		developer	years
		products and services based	creation
		on open data.
	B	Development		and	Big data	11-50	Data / IT	About a year
		improvement	of	new	consultant		consulting
		products and services based	& engineer
		on open data.
	C	Find and compare all Owner	1-10	App	3-5 years

See the report by the Dutch Court of Audit (2016) on trends in open data, p. 42, available at http://www.rekenkamer.nl/dsresource?objectid=23808&type=org

employees from a variety of organizations and domains may inspire innovation. Furthermore it is essential that data users and suppliers keep discussing their data needs and ensure that open data is more reliable and usable for external stakeholders, decreasing the risks for data innovators regarding data quality and availability.