Andreas Holzinger
email: a.holzinger@hci-kdd.org

Markus Plass
email: m.plass@hci-kdd.org

Katharina Holzinger
email: k.holzinger@hci-kdd.org

Gloria Cerasela Crişan

Camelia-M Pintea
email: dr.camelia.pintea@ieee.org

Vasile Palade
email: vasile.palade@coventry.ac.uk

Towards interactive Machine Learning (iML): Applying Ant Colony Algorithms to Solve the Traveling Salesman Problem with the Human-in-the-Loop Approach

Keywords: interactive Machine Learning, Human-in-the-Loop, Traveling Salesman Problem, Ant Colony Optimization

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction and Motivation for Research

Automatic Machine Learning (aML) is increasingly making big theoretical as well as practical advances in many application domains, for example, in speech recognition [START_REF] Dong | Guest editorial: Advances in machine learning for speech processing[END_REF], recommender systems [START_REF] Aggarwal | Ensemble-based and hybrid recommender systems[END_REF], or autonomous vehicles [START_REF] Sofman | Improving robot navigation through self-supervised online learning[END_REF].

The aML-approaches sometimes fail or deliver unsatisfactory results, when being confronted with complex problem. Here interactive Machine Learning (iML) may be of help and a "human-in-the-loop" may be beneficial in solving computationally hard problems, where human expertise can help to reduce, through heuristics, an exponential search space.

We define iML-approaches as algorithms that can interact with both computational agents and human agents and can optimize their learning behaviour through these interactions [START_REF] Holzinger | Interactive machine learning (iml)[END_REF]. To clearly distinguish the iML-approach from a classic supervised learning approach, the first question is to define the human's role in this loop (see Figure 1), [START_REF] Holzinger | Interactive machine learning for health informatics: When do we need the human-in-the-loop?[END_REF].

Fig. 1. The iML human-in-the-loop approach: The main issue is that humans are not only involved in pre-processing, by selecting data or features, but actually during the learning phase they are directly interacting with the algorithm, thus shifting away the black-box approach to a glass-box; there might also be more than one human agent interacting with the computational agent(s), allowing for crowdsourcing or gamification approaches There is evidence that humans sometimes still outperform ML-algorithms, e.g. in the instinctive, often almost instantaneous interpretation of complex patterns, for example, in diagnostic radiologic imaging: A promising technique to fill the semantic gap is to adopt an expert-in-the-loop approach, by integrating the physicians high-level expert knowledge into the retrieval process and by acquiring his/her relevance judgments regarding a set of initial retrieval results [START_REF] Akgul | Content-based image retrieval in radiology: Current status and future directions[END_REF]. Despite these apparent assumption, so far there is little quantitative evidence on effectiveness and efficiency of iML-algorithms. Moreover there is practically no evidence of how such interaction may really optimize these algorithms as it is a subject that is still being studied by cognitive scientists for quite a while and even though "natural" intelligent agents are present in large numbers throughout the world [START_REF] Gigerenzer | Heuristic decision making[END_REF].

From the theory of human problem solving it is known that for example, medical doctors can often make diagnoses with great reliability -but without being able to explain their rules explicitly. Here iML could help to equip algorithms with such "instinctive" knowledge and learn thereof. The importance of iML becomes also apparent when the use of automated solutions due to the incompleteness of ontologies is difficult [START_REF] Atzmueller | Introspective subgroup analysis for interactive knowledge refinement[END_REF].

This is important as many problems in machine learning and health informatics are N P-hard, and the theory of N P-completeness has crushed previous hopes that N P-hard problems can be solved within polynomial time [START_REF] Papadimitriou | Computational complexity[END_REF]. Moreover, in the health domain it is sometimes better to have an approximate solution to a complex problem, than a perfect solution to a simplified problem -and this within a reasonable time, as an average medical doctor has on average less than five minutes to make a decision [START_REF] Gigerenzer | Gut Feelings: Short Cuts to Better Decision Making[END_REF].

Consequently, there is much interest in approximation and heuristic algorithms that can find near optimal solutions within reasonable time. Heuristic algorithms are typically among the best strategies in terms of efficiency and solution quality for problems of realistic size and complexity. Meta-heuristic algorithms are widely recognized as one of the most practical approaches for combinatorial optimization problems. Some of the most useful meta-heuristic algorithms include genetic algorithms, simulated annealing and Ant Colony Optimization (ACO).

In this paper we want to address the question whether and to what extent a human can be beneficial in direct interaction with an algorithm. For this purpose we developed an online-platform in the context of our iML-project [START_REF] Holzinger | iML online[END_REF], to evaluate the effectiveness of the "human-in-the-loop" approach and to discuss some strengths and weaknesses of humans versus computers. Such an integration of a "human-into-the-loop" may have many practical applications, as e.g. in health informatics, the inclusion of a "doctor-into-the-loop" [START_REF] Kieseberg | A tamper-proof audit and control system for the doctor in the loop[END_REF], [START_REF] Kieseberg | Trust for the doctor-in-the-loop[END_REF] can play a significant role in support of solving hard problems, particularly in combination with a large number of human agents (crowdsourcing).

Background

Problem Solving: Human versus Computer

Many ML-methods perform very badly on extrapolation problems which would be very easy for humans. An interesting experiment was performed by [START_REF] Wilson | The human kernel[END_REF]: humans were presented functions drawn from Gaussian Processes (GP) with known kernels in sequence and asked to make extrapolations. The human learners extrapolated on these problems in sequence, so having an opportunity to progressively learn about the underlying kernel in each set. To further test progressive function learning, they repeated the first function at the end of the experiment, for six functions in each set. The authors asked for extrapolation judgements because it provides more information about inductive biases than interpolation and pose difficulties for conventional GP kernels [START_REF] Wilson | Gaussian process kernels for pattern discovery and extrapolation[END_REF]. Research in this area, i.e. at the intersection of cognitive science and computational science is fruitful for further improving aML thus improve performance on a wide range of tasks, including settings which are difficult for humans to process (for example big data and high dimensional problems); on the other hand such experiments may provide insight into brain informatics.

The first question is: When does a human still outperform ML-algorithms? ML algorithms outperform humans for example in high-dimensional data processing, in rule-based environments, or in automatic processing of large quantities of data (e.g. image optimization). However, ML-algorithms have enormous problems when lacking contextual information, e.g. in natural language translation/curation, or in solving N P-hard problems. One important issue is in socalled unstructured problem solving: Without a pre-set of rules, a machine has trouble solving the problem, because it lacks the creativity required for complex problem solving. A good example for the literal competition of the human mind and its supposed artificial counterpart are various games, because they require human players to use their skill in logic, strategic thinking, calculating or creativity. Consequently, it is a good method to experiment on the strength and weaknesses of both brains and algorithms. The field of ML actually started with such efforts: In 1958 the first two programs to put the above question to the test were a checker program by Arthur Samuel and the first full chess program by Alex Bernstein [START_REF] Bernstein | A chess playing program for the ibm 704[END_REF]. Whilst Samuel's program managed to beat Robert Nealey, the Connecticut checkers champion at that time, chess proved to be the computers weakness at that time; because on average just one move offers a choice of 30 possibilities, with an average length of 40 moves that leaves 10 120 possible games. Recently, computers had impressive results in competitions against humans: In 1997, the world chess champion Garry Kasparov lost a six-game match against Deep Blue. A more recent example is the 2016 Google DeepMind Challenge, a five-game match between the world Go champion Lee Sedol and AlphaGo, developed by the Google DeepMind team. Although AlphaGo won the overall game, it should be mentioned that Lee Sedol won one game. These examples just shall demonstrate how much potential a combination of both sides may offer [START_REF] Holzinger | Human-computer interaction & knowledge discovery (HCI-KDD): What is the benefit of bringing those two fields to work together?[END_REF].

As test case for our approach we selected the Traveling Salesman Problem, which is a classical hard problem in computer science and studied for a long time, and where Ant Colony Optimization has been used to provide approximate solutions [START_REF] Crişan | Ant-based system analysis on the traveling salesman problem under real-world settings[END_REF].

Traveling Salesman Problem (TSP)

The TSP appears in a number of practical problems in health informatics, e.g. the native folded three-dimensional conformation of a protein is its lowest free energy state and both a two-and three-dimensional folding processes as a free energy minimization problem belong to a large set of computational problems, assumed to be very hard (conditionally intractable) [START_REF] Crescenzi | On the complexity of protein folding[END_REF].

The TSP basically is about finding the shortest path through a set of points, returning to the origin. As it is an intransigent mathematical problem, many heuristics have been developed in the past to find approximate solutions [START_REF] Macgregor | Human performance on the traveling salesman problem[END_REF].

Numerical examples of real-world TSP tours are given in [START_REF] Crişan | An analysis of the hardness of novel tsp iberian instances[END_REF]; so, for example in Sweden for 24,978 cities the length is approximative 72,500 km [START_REF] Cook | TSP online: www.math.uwaterloo.ca/tsp[END_REF] and in Romania: for 2950 cities, the length is approximative 21,683 km) [START_REF] Crişan | Emergency management using geographic information systems: application to the first romanian traveling salesman problem instance[END_REF]. The World TSP tour, with the length 7,516,353,779 km was obtained by K. Helsgaun using 1,904,711 cities [START_REF] Cook | TSP online: www.math.uwaterloo.ca/tsp[END_REF].

The Mathematical Background of TSP The TSP is a important graphbased problem which was firstly claimed to be a mathematical problem in 1930 [START_REF] Cook | In pursuit of the traveling salesman: mathematics at the limits of computation[END_REF]. Given a list of cities and their pairwise distances: find the shortest possible tour that visits each city exactly once. It is a N P-hard problem, meaning that there is no polynomial algorithm for solving it to optimality. For a given number of n cities there are (n -1)! different tours.

In terms of integer linear programming the TSP is formulated as follows [START_REF] Dantzig | Linear programming and extensions[END_REF][START_REF] Papadimitriou | Combinatorial optimization: algorithms and complexity[END_REF][START_REF] Tucker | On directed graphs and integer programs[END_REF].

The cities, as the nodes, are in the set N of numbers 1, . . . , n; the edges are

L = {(i, j) : i, j ∈ N , i = j}
There are considered several variables: x ij as in equation ( 1), the cost between cities i and j denoted with c ij .

x ij =
1 , the path goes from city i to city j 0 otherwise

The Traveling Salesman Problem is formulated to optimize, more precisely to minimize the objective function illustrated in equation ( 2).

min n i=1 n i =j,j=1 c ij x ij (2)
The TSP constraints follow.

-The first condition, equation (3) is that each node i is visited only once.

i∈N ,(i,j)∈L

x ij + j∈N ,(i,j)∈L x ji = 2 (3)
-The second condition, equation ( 4), ensures that no subtours, S are allowed.

i,j∈L,(i,j)∈S

x ij ≤ |S| -1, ∀S ⊂ N : 2 ≤ |S| ≤ n -2 (4)
For the symmetric TSP the condition c ij = c ji holds. For the metric version the triangle inequality holds:

c ik + c kj ≥ c ij , ∀i, j, k nodes.

Ant Algorithms

There are many variations of the Ant Colony Optimization applied on different classical problems. For example an individual ant composes a candidate solution, beginning with an empty solution and adding solution components iteratively until a final candidate solution has been generated [START_REF] Dorigo | Ant colony optimization -artificial ants as a computational intelligence technique[END_REF]. The ants solutions are not guaranteed to be optimal and hence may be further improved using local search methods. Based on this observation, the best performance is obtained using hybrid algorithms combining probabilistic solution constructed by a colony of ants with local search algorithms as 2-opt, 3-opt, tabu-search etc. In hybrid algorithms, the ants can be seen as guiding the local search by constructing promising initial solutions. Conversely, the local search guides the colony evolution, because ants preferably use solution components which, earlier in the search, have been contained in good locally optimal solutions.

Ant Behaviour and Pheromone trails

Ants are (similar as termites, bees, wasps) socially intelligent insects living in organized colonies where each ant can communicate with each other. Pheromone trails laid by foraging ants serve as a positive feedback mechanism for the sharing of information. This feedback is nonlinear, in that ants do not react in a proportionate manner to the amount of pheromone deposited, instead, strong trails elicit disproportionately stronger responses than weak trails. Such nonlinearity has important implications for how an ant colony distributes its workforce, when confronted with a choice of food sources [START_REF] Sumpter | From nonlinearity to optimality: pheromone trail foraging by ants[END_REF].

This leads to the emergence of shortest paths and when an obstacle breaks the path, ants try to get around the obstacle randomly choosing either way. If the two paths encircling the obstacle have the different length, more ants pass the shorter route on their continuous pendulum motion between the nest points in particular time interval. While each ant keeps marking its way by pheromone the shorter route attracts more pheromone concentrations and consequently more and more ants choose this route. This feedback finally leads to a stage where the entire ant colony uses the shortest path. Each point at which an ant has to decide which solution component to add to its current partial solution is called a choice point.

The ant probabilistically decides where to go by favouring the closer nodes and the nodes connected by edges with higher pheromone trails. After the solution construction is completed, the ants give feedback on the solutions they have constructed by depositing pheromone on solution components which they have used in their solution. Solution components which are part of better solutions or are used by many ants will receive a higher amount of pheromone and, hence, will more likely be used by the ants in future iterations of the algorithm. To avoid the search getting stuck, typically before the pheromone trails get reinforced, all pheromone trails are decreased by a factor [START_REF] Dorigo | Ant colony optimization: overview and recent advances[END_REF], [START_REF] Li | Chaosorder transition in foraging behavior of ants[END_REF].

Such principles inspired from observations in nature can be very useful for the design of multi-agent systems aiming to solve hard problems such as the TSP. Pioneer work in that respect was done by Marco Dorigo, the inventor of the Ant Colony Optimization (ACO) meta-heuristic for combinatorial optimization problems [START_REF] Colorni | Distributed optimization by ant colonies[END_REF].

In summary it can be stated that ACO algorithms are based on simple assumptions:

-Foraging ants construct a path in a graph and some of them (according to updating rules) lay pheromone trail on their paths -Decisions are based on pheromone deposited on the available edges and on the distance to the available nodes -Pheromone evaporates over time -Every ant remembers already visited places -Foragers prefer to follow the pheromone trails and to choose close nodes -Local information improve the information content of pheromone trails -The colony converges to a high quality solution.

Convergence is a core-competence of distributed decision-making by insect colonies; an ant colony operates without central control, regulating its activities through a network of local interactions [START_REF] Gordon | The rewards of restraint in the collective regulation of foraging by harvester ant colonies[END_REF]. The evolution in this case is a probabilistic stepwise construction of a path, making use of pheromones and problem-specific heuristic information to incrementally find a solution [START_REF] Yang | Nature-inspired optimization algorithms[END_REF], [START_REF] Brownlee | Clever algorithms: nature-inspired programming recipes[END_REF].

ACO Procedure Ant Colony Optimization (ACO) metaheuristic is a framework for solving combinatorial optimization problems. Depending on the specific problem tackled, there are many successful ACO realizations. One of the oldest and specifically dedicated to TSP is the Ant Colony System (ACS) [START_REF] Dorigo | Ant colony system: A cooperative learning approach to the traveling salesman problem[END_REF]. In ACS, ants concurrently traverse a complete graph with n nodes, construct solutions and deposit pheromone on their paths. The distances between nodes are stored in the matrix (d ij ), and the pheromone on edges are in (τ ij ).

The pseudocode of Ant Colony Systems is illustrated in Algorithm 1. The characteristics of ACS are:

the decision rule for an ant staying in node i for choosing the node j is a mix of deterministic and probabilistic processes. two rules define the process of pheromone deposit.

The most important ACS parameters are: the number of ants (m), the balance between the effect of the problem data and the effect of the pheromone (β), the threshold for deterministic decisions (q 0 ) and the evaporation rate of the pheromone (ρ). These parameters allow the following description of ACS.

At the beginnig, a TSP solution is generated, using a heuristic method (for example, the Nearest Neighbor). At this step, this solution is considered the global best. The ants are deployed at random in the nodes and they move to other nodes, until they complete a solution. If an ant stays in the node i, it can move to one of the unvisited nodes. The available nodes form the set J i . The next node j is chosen based on the pheromone quantity on the corresponding edge, on the edge's length, and on an uniformly generated random value q∈ [0, 1]. η is considered the inverse of the distance between two nodes. If q ≤ q 0 , then the equation ( 5) holds. Otherwise, j is randomly selected from the available nodes using a proportional rule, based on the probabilities (equation 6).

j = argmax l∈Ji (τ il • [η il ] β ) (5)
p ij = τ ij • [η ij ] β l∈Ji τ il • [η il ] β (6)
After all the solutions are constructed, their lengths are computed, and the global best is updated, if a better solution is founded. The local pheromone update is applied. Each ant updates the pheromone on its path using equation [START_REF] Gigerenzer | Heuristic decision making[END_REF].

τ ij (t + 1) = (1 -ρ) • τ ij (t) + ρ 1 n • L initial (7)
where L initial is the length of the initial tour. The current iteration of the ACS is ended by applying the global pheromone update rule: Only the current best ant reinforces the pheromone on its path, using equation [START_REF] Atzmueller | Introspective subgroup analysis for interactive knowledge refinement[END_REF].

τ ij (t + 1) = (1 -ρ) • τ ij (t) + ρ 1 L best ( 8
)
where L best is the length of the best tour. The algorithm is repeated until the stopping conditions are met and exits with the best solution.

Inner Ant System for TSP

In [START_REF] Pintea | Combining heuristics and modifying local information to guide ant-based search[END_REF] the Inner Ant System (IAS) is introduced, also known as Inner Update System in [START_REF] Pintea | Improving ant systems using a local updating rule[END_REF] where the "inner" rule was firstly introduced to reinforce the local search during an iteration. The structure of the IAS is similar with the Ant Colony System. After the inner rule the Lin-Kernighan 2-opt and 3-opt [START_REF] Helsgaun | An effective implementation of the lin-kernighan traveling salesman heuristic[END_REF] are used to improve the local solution.

After each transition the trail intensity is updated using the inner correction rule, equation [START_REF] Papadimitriou | Computational complexity[END_REF], from [START_REF] Pintea | Improving ant systems using a local updating rule[END_REF].

τ ij (t + 1) = (1 -ρ)τ ij (t) + ρ 1 n • L + (9)
where L + is the cost of the best tour.

In Ant Colony Systems only ants which generate an optimal tour are allowed to globally update the pheromone. The global update rule is applied to the edges belonging to the best tour. The correction rule is equation [START_REF] Atzmueller | Introspective subgroup analysis for interactive knowledge refinement[END_REF]. In IAS the pheromone trail is over an upper bound τ max , the pheromone trail is re-initialized as in [START_REF] Stuetzle | Max-min ant system and local search for the traveling salesman problem[END_REF]. The pheromone evaporation is used after the global pheromone update rule.

Experimental Method, Setting and Results

The ACO as an aML algorithm usually use no interaction. The ants walk around and update the global weights after each iteration. This procedure is repeated a distinct number of times. Following the iML-approach the human now can open the black-box and can manipulate this algorithm by changing the behavior of the ants. This is done by changing the pheromones, i.e. the human has the possibility to add or remove pheromones after each iteration.

Experimental Method

Based on the Inner Ant System for TSP this could be understood as adding or removing of pheromones on a track between cities. So the roads become more or less interesting for the ants and and there is a high chance that they will consider road changes. In pseudocode we can write Algorithm 2.

For testing the Traveling Salesman Problem using iML we implemented an online tool with the following workflow:

-→ Click on the empty field to add new cities.

-→ Press "Start" to initialize the ants and to let them run.

-→ With a click on "Pause/Resume" the algorithm will be paused.

-→ Selection of two edges (first vertex and second vertex).

-→ Between these two edges the pheromones can be now adjusted by the slider below.

-→ With "Set Pheromone" changes of pheromones are written in the graph.

-→ Another click on "Pause/Resume" continues the algorithm.

-→ The steps above can be repeated as often as needed.

Implementation Details

The implementation is based on Java-Script. So it is a browser based solution which has the great benefit of platform independence and no installation is required.

For the test-setup we used 30 ants, 250 iterations. For the other parameters we choose fixed default values: α =1; β =3; ρ =0.1. The parameters are in the current prototype fixed, this makes it easier to compare the results.

When starting the application an empty map appears on the left side. By clicking on the map a new city is created. The script automatically draws the connections between all the cities. After starting the algorithm, a list of cities/numbers will appear in the control center (see figure 2).

After selecting two of them, the pheromones on the track between the cities can be adjusted with the slider. The current amount of pheromones is displayed as blue line with variance in the width for the amount of pheromones. After each iteration the current shortest path is calculated. If the new path is shorter than the old one, the green line will be updated. For the evaluation on testing process there are some pre-defined data sets. From these data sets the best solution is known. The original data sets can be found on [START_REF] Reinelt | TSPLIB -Library of sample instances for the TSP[END_REF].

The results of the pre-defined datasets can be compared with the optimal solution after finishing the algorithm by clicking on "Compare with optimal tour". The optimal tour is displayed with a red line. It is also possible to extend the existing data sets by adding points. A deletion of points is currently not possible, but we are working on it, as this allows a lot of interesting experiments (perturbation of graph structures). Saving and loading modules brings the benefit that the user can store and share the results if he discovered something interesting. During the loading the saved data is maximized to the size of the window.

Experimental Results

The update of the pheromones on a path changes the ant's behavior. Because of human interaction such as adding pheromones the chances that an ant might take that path become higher. Consequently, this is a direct interaction with the algorithm. The interaction could not be effective through the evaporation of pheromones if this happens the user has to set pheromones again.

The interaction only changes the probability that the path could be taken. It has no effect on the shortest path. So it is obvious that a change of the pheromones may not necessarily result in a better path. An increase of this probability that it has an effect can be done by increasing the numbers of Iterations. If we take a closer look at our data set "14 cities in Burma", Burma14.tsp, it is possible to see this effect. The path between city 1 and 10 is part of the shortest path (see figure 3).

If we start the algorithm without interaction the probability that the shortest path is taken is very low. If we now increase some of our parameters like colony size (from 20 to 40) and the number of iterations (from 250 to 500),the chances become higher, that the algorithm will return the shortest path. When we now do the same with human interaction (add pheromones to path 1-10 and decrease pheromones on the surrounding paths), after iterations 10, 60, 100 and 150, then there is a high probability that the shortest path can be found in 250 iterations (see figure 4). Fig. 3. Not optimal path of Burma14.tsp dataset using an ant algorithm without iML after 250 iterations and using twenty ants. As we see in the test results above, the iML-approach poses benefits, however, further evaluations must also answer questions regarding the cost, reliability, robustness and subjectivity of humans-in-the-loop. Further studies must answer questions including: "When is it going to be inefficient?" In the example above the number of iterations can be reduced from 500 to 250, by three human interactions. The benefit is that after a few iterations the user can intuitively observe whether and to what extent ants are on a wrong way. The best number of cities, for an intuitively decision, is between about 10 and 150. A number beyond 150 cities poses higher complexity and prevent a global overview. An increase of the cities is possible, if there is for example a preprocessing phase of subspace clustering [START_REF] Hund | Visual analytics for concept exploration in subspaces of patient groups: Making sense of complex datasets with the doctor-in-the-loop[END_REF].

Further challenges are in the transfer of our approach to other nature inspired algorithms which have a lot of potential for the support of solving hard problems [START_REF] Holzinger | Darwin or lamarck? future challenges in evolutionary algorithms for knowledge discovery and data mining[END_REF].

iML can be particular helpful whenever we are lacking large amounts of data, deal with complex data and/or rare events where traditional aML-algorithms suffer due to insufficient training samples. A doctor-in-the-loop can help where human expertise and long-term experience is useful, however, the optimal way will be in hybrid solutions in combining the "best of two worlds", following the HCI-KDD approach [START_REF] Holzinger | Trends in interactive knowledge discovery for personalized medicine: Cognitive science meets machine learning[END_REF][START_REF] Holzinger | Human-computer interaction & knowledge discovery (HCI-KDD): What is the benefit of bringing those two fields to work together?[END_REF].

In the future it is planned to use gamification [START_REF] Ebner | Successful implementation of user-centered game based learning in higher education: An example from civil engineering[END_REF] and crowdsourcing [START_REF] Raykar | Learning from crowds[END_REF], with the grand goal to make use of many humans to help to solve hard problems.

Conclusion

We demonstrated that the iML approach [START_REF] Holzinger | Interactive machine learning for health informatics: When do we need the human-in-the-loop?[END_REF] can be used to improve current TSP solving methods. With this approach we have enriched the way the humancomputer interaction is used [START_REF] Amershi | Power to the people: The role of humans in interactive machine learning[END_REF]. This research is in-line with other successful approaches. Agile software paradigms, for example, are governed by rapid responses to change and continuous improvement. The agile rules are a good example on how the interaction between different teams can lead to valuable solutions. Gamification seeks for applying the Games Theory concepts and results to non-game contexts, for achieving specific goals. In this case, gamification could be helpful by considering the human and the computer as a coalition. There are numerous open research directions. The challenge now is to translate these approach to other similar problems, for example on protein folding, and at the same time to scale up on complex software applications.

Fig. 2 .

 2 Fig. 2. The control-panel of our Java-Script code

Fig. 4 .

 4 Fig. 4. Optimal path of Burma14.tsp dataset using iML, after 250 iterations including four human interactions and twenty ants for the ant algorithm.

Algorithm 1 :

 1 Ant Colony System Algorithm

	Input : ProblemSize, m, β, ρ, σ, q0
	Output: Pbest
	Pbest ← CreateHeuristicSolution(ProblemSize);
	Pbestcost ← Cost(Pbest);
	Pheromoneinit ←	1.0 P roblemSize×P best cost ;
	P heromone ← InitializePheromone(Pheromoneinit);
	while ¬StopCondition() do
	for i = 1 to m do
	Si ← ConstructSolution(Pheromone, ProblemSize, β, q0);
	Sicost ← Cost(Si);
	if Sicost ≤ Pbestcost then
	Pbestcost ← Sicost;
	Pbest ← Si;
	end
	LocalUpdateAndDecayPheromone(Pheromone, Si, Sicost, ρ);
	end
	GlobalUpdateAndDecayPheromone(Pheromone, Pbest, Pbestcost, ρ);
	end
	return Pbest;

  Algorithm 2: Ant Colony Algorithm iML

	Input : ProblemSize, m, β, ρ, σ, q0
	Output: Pbest
	Pbest ← CreateHeuristicSolution(ProblemSize);
	Pbestcost ← Cost(Pbest);
	Pheromoneinit ←	1.0 P roblemSize×P best cost ;
	P heromone ← InitializePheromone(Pheromoneinit);
	while ¬StopCondition() do
	for i = 1 to m do
	Si ← ConstructSolution(Pheromone, ProblemSize, β, q0);
	Sicost ← Cost(Si);
	if Sicost ≤ Pbestcost then
	Pbestcost ← Sicost;
	Pbest ← Si;
	end
	LocalUpdateAndDecayPheromone(Pheromone, Si, Sicost, ρ);
	end
	GlobalUpdateAndDecayPheromone(Pheromone, Pbest, Pbestcost, ρ);
	while isUserInteraction() do
	GlobalAddAndRemovePheromone(Pheromone, Pbest, Pbestcost, ρ);
	end
	end
	return P best ;


Inger Anne Tøndel
email: inger.a.tondel@sintef.no

Erlend Fredrik Seehusen

Andreas Gjaere

Marie Elisabeth

Gaup Moe

Differentiating cyber risk of insurance customers: the insurance company perspective

Keywords: cyber-insurance, risk management, risk modeling Control evaluation and implementation Control identification Risk determination Threat and vulnerability assessment System characterisation

As a basis for offering policy and setting tariffs, cyber-insurance carriers need to assess the cyber risk of companies. This paper explores the challenges insurance companies face in assessing cyber risk, based on literature and interviews with representatives from insurers. The interview subjects represent insurance companies offering cyber-insurance in a market where this is a new and unknown product. They have limited historical data, with few examples of incidents leading to payout. This lack of experience and data, together with the need for an efficient sales process, highly impacts their approach to risk assessment. Two options for improving the ability to perform thorough yet efficient assessments of cyber risk are explored in this paper: basing analysis on reusable sectorspecific risk models, and including managed security service providers (MSSPs) in the value chain.

Introduction

Cyber-insurance has been defined in literature as "the transfer of financial risk associated with network and computer incidents to a third party" [START_REF] Böhme | Modeling cyber-insurance: Towards a unifying framework[END_REF]. It can take many forms, offering third party or first party coverage, and covering a variety of threat types [START_REF] Biener | Insurability of cyber risk: An empirical analysis[END_REF][START_REF] Marotta | A survey on cyberinsurance[END_REF]. The demand for this insurance product is increasing [START_REF] Perlroth | Cyberattack insurance a challenge for business[END_REF][START_REF]Managing cyber security as a business risk: Cyber insurance in the digital age[END_REF]. Although cyber-insurance has been around in some form for several decades, the cyber-insurance products are still relatively immature. This is underlined by statements such as "cyber policies are still the Wild West of insurance policies" [START_REF] Chickowski | 10 things IT probably doesn't know about cyber insurance[END_REF] and "products are untested, pricing appears arbitrary and experimentation in contract writing is commonplace" [START_REF] Bandyopadhyay | Towards a managerial decision framework for utilization of cyber insurance instruments in IT security[END_REF]. Academic research on cyber-insurance has identified a number of challenges and knowledge gaps [START_REF] Bandyopadhyay | Towards a managerial decision framework for utilization of cyber insurance instruments in IT security[END_REF][START_REF] Marotta | A survey on cyberinsurance[END_REF][START_REF] Tøndel | Using cyberinsurance as a risk management strategy: Knowledge gaps and recommendations for further research[END_REF], some of which are related to assessing cyber risk.

Taking on cyber in their product portfolio is associated with a greater risk for insurance companies than other traditional covers, which is reflected in the product's pricing. According to a UK study, the cost of cyber-insurance relative to the limit purchased is typically three times the cost of cover for more established general liability risks, and six times higher than for property insurance [START_REF] Ltd | UK cyber security: The role of insurance in managing and mitigating the risk[END_REF]. The UK study additionally points out that cyber-insurance has a lower price differentiation across customers, something that may be due to a lack of historical data in underwriting or inappropriate means of assessing the cyber risk of potential customers. This is concerning as it undermines the role that insurance can have in increasing the security posture of insurance buyers, since they will not see any benefit in terms of lower insurance cost [START_REF] Ltd | UK cyber security: The role of insurance in managing and mitigating the risk[END_REF].

As a basis for offering policy and setting tariffs, cyber-insurance carriers need to assess the cyber risk of companies. Insurance companies do this to differentiate between potential clients, thus reducing the risk of adverse selection [START_REF] Pal | On differentiating cyber-insurance contracts a topological perspective[END_REF]. This paper explores the challenges insurance companies face in assessing cyber risk, based on literature (Section 2) and interviews with representatives from insurance companies (Section 3). Section 4 outlines two options for improving the ability to perform thorough, yet efficient assessments of cyber risk: basing analysis on reusable sector-specific risk models, and including managed security service providers (MSSPs) in the value chain. Section 5 discusses the contribution of the paper and provides suggestions for further work. Section 6 concludes the paper.

Known challenges for assessing cyber risk of insurance customers

A large number of standards, guidelines and research papers suggest methods for information security risk assessments [START_REF] Sulaman | A review of research on risk analysis methods for IT systems[END_REF]. Though they have their differences, the methods tend to include similar steps: characterisation of the system; threat and vulnerability assessment; risk determination; control identification, and; evaluation and implementation of controls [START_REF] Fenz | Verification, validation, and evaluation in information security risk management[END_REF]. Fig. 1 provides an overview of the risk assessment process and key challenges for insurance companies, identified through searches in academic literature as well as non-academic sources, such as news articles, technical reports and white papers (see Tøndel et al. [START_REF] Tøndel | Using cyberinsurance as a risk management strategy: Knowledge gaps and recommendations for further research[END_REF] for more details on the method used for the literature study).

The cyber risk of an organisation depends on various internal and external conditions, including the organisation's assets, the technology they are using and its vulnerabilities, the security awareness and competence of the employees, routines relevant for cyber security, the security of the organisation's vendors, vulnerabilities in common infrastructure which the organisation relies on, and the motivation of potential attackers. With all these factors to consider, and limited knowledge of the impact of the various factors on the organisation's overall risk, risk is complex to understand and evaluate, and it is impossible to verify that the risk estimation is correct [START_REF] Marotta | A survey on cyberinsurance[END_REF]. This is true for the organisation itself, but also for an insurance company offering cyber-insurance to the organisation. The premium paid by all insurance holders should cover any payouts plus return profit to the insurance company. To limit their own risk exposure, the insurers seek a mix of customers which provide a sufficient premium income compared to the overall risk portfolio, and a steady flow of payouts. The question of what constitutes "good IT security" has not been answered conclusively [START_REF]ENISA: Incentives and barriers of the cyber insurance market in europe[END_REF]. Research and practice on measuring information security has progressed, and there are many indicators and measurement frameworks available, see e.g. Herrmann [START_REF] Herrmann | Complete Guide to Security and Privacy Metrics[END_REF] and ISO/IEC 27004 [20]. Still, there is no agreed set of metrics to predict information security risk in the general case [START_REF] Toregas | Insurance for cyber attacks: The issue of setting premiums in context[END_REF]. Setting security baselines, or providing rewards for companies with documented security best practices, is difficult due to lack of knowledge about the effect of different security controls [START_REF]Cybersecurity insurance workshop readout report[END_REF]. Simple metrics, like the number of records lost, does not always correlate with the total cost of the breach [START_REF]NetDilgence: Netdiligence cyber claims study 2014[END_REF]. Currently, different insurance carriers evaluate risk in different ways, and they consider a variety of risk factors, covering both technical and organisational aspects [START_REF]Cybersecurity insurance workshop readout report[END_REF][START_REF]Cyber risk culture roundtable readout report[END_REF].

The lack of robust actuarial data has been pointed out by various sources as a reason for limited success of the cyber-insurance market [START_REF] Bandyopadhyay | Why IT managers don't go for cyber-insurance products[END_REF][START_REF]ENISA: Incentives and barriers of the cyber insurance market in europe[END_REF][START_REF] Ey | Mitigating cyber risk for insurers, part 2: Insights into cyber security and risk[END_REF][START_REF] Toregas | Insurance for cyber attacks: The issue of setting premiums in context[END_REF]. Several sources of historical cyber-incident information exist, e.g. from CERTs, security companies or researchers [START_REF]ENISA: Incentives and barriers of the cyber insurance market in europe[END_REF]. Examples of surveys that provide relevant data on costs of cyber-incidents are a NetDiligence survey of insurance payouts related to cyber liability [START_REF]NetDilgence: Netdiligence cyber claims study 2014[END_REF] and Ponemon's Cost of Data Breach Study [START_REF]Ponemon: 2014 cost of data breach study: Global analysis[END_REF]. However, it is not easy to determine which sources of information should be relied upon more than others [START_REF]ENISA: Incentives and barriers of the cyber insurance market in europe[END_REF]. Barriers for information sharing include reluctance by firms to reveal details on security incidents [START_REF] Bandyopadhyay | Why IT managers don't go for cyber-insurance products[END_REF][START_REF] Gordon | A framework for using insurance for cyberrisk management[END_REF][START_REF] Toregas | Insurance for cyber attacks: The issue of setting premiums in context[END_REF], and limited ability to quantify costs associated with such incidents [START_REF] Toregas | Insurance for cyber attacks: The issue of setting premiums in context[END_REF]. Toregas and Zahn make the following claim: "Given that many companies are either unaware of a cyber attack or unwilling to disclose such attacks, and added to the fact that those attacks are hard to quantify, actuarial data for the cyber-insurance market is missing and unlikely to be available in the near future" [START_REF] Toregas | Insurance for cyber attacks: The issue of setting premiums in context[END_REF]. Insurance carriers are also reluctant to share incident information, due to a competitive market, and because they fear that they would ultimately "give more than they get" [START_REF]Cybersecurity insurance workshop readout report[END_REF]. The significant information asymmetries currently present require insurance companies to perform costly state verification and upfront risk assessments [START_REF] Biener | Insurability of cyber risk: An empirical analysis[END_REF].

Carriers engage clients heavily during the underwriting process. Historically they used extensive questionnaires [START_REF] Baer | Cyberinsurance in IT security management[END_REF][START_REF]Cyber risk culture roundtable readout report[END_REF], but it is becoming more common to speak directly with clients to understand their vulnerabilities and risk management controls [START_REF]Cyber risk culture roundtable readout report[END_REF]. In this process customers may need to share a potentially large amount of information with the insurance company. It has been pointed out that in a competitive environment, potential customers may favour insurance companies with less demanding assessment processes [START_REF]Cybersecurity insurance workshop readout report[END_REF]. Likewise, potential clients may not like being dictated by the insurance company on how to mitigate cyber risks [START_REF]Cyber insurance roundtable readout report -health care and cyber risk management: Cost/benefit approaches[END_REF]. Insurance companies are however also interested in having an efficient assessment process, illustrated by quotes such as "carriers typically don't spend weeks with potential insureds reviewing every single aspect of an organization to see what's happening with its implementation of information security policies" [START_REF]Cybersecurity insurance workshop readout report[END_REF] and "cybersecurity insurance underwriting essentially tries to weed out the 20 percent of companies who have no clue about cybersecurity from the pool of potential insureds" [START_REF]Cyber risk culture roundtable readout report[END_REF]. Despite this need to have an efficient process, from the viewpoints of both customers and providers, it should be pointed out that the upfront risk assessment performed by insurance companies can have positive side-effects on increased self-protection, and the consulting and risk assessment services provided by the insurance company is one central driver of product value [START_REF] Biener | Insurability of cyber risk: An empirical analysis[END_REF].

The experienced cyber risk may change rapidly based on technology changes, discovery of vulnerabilities, political actions etc. There is a need to understand how to take these changes into account when it comes to cyber-insurance. Organisational resilience, i.e. the capability of recognizing, adapting to and coping with the unexpected [START_REF] Woods | Essential characteristics of resilience. Resilience engineering: concepts and precepts[END_REF] is relevant for this. In the safety domain, research has progressed on measuring organisational resilience through risk awareness, response capacity and support [START_REF] Øien | Development of early warning indicators based on resilience engineering[END_REF], and such a measurement framework has been adapted to the ICT domain [START_REF] Bernsmed | Forewarned is forearmed: Indicators for evaluating information security incident management[END_REF]. Rapid changes are additionally a challenge for collecting reliable actuarial data, as changes in technology and attacker profiles can cause empirical information on incidents to quickly become outdated [START_REF] Biener | Insurability of cyber risk: An empirical analysis[END_REF][START_REF] Marotta | A survey on cyberinsurance[END_REF][START_REF]Cybersecurity insurance workshop readout report[END_REF].

According to Böhme and Schwartz [START_REF] Böhme | Modeling cyber-insurance: Towards a unifying framework[END_REF], cyber risk is characterised by both interdependent security and correlated risk; the security of a node is dependent on the security of other nodes and incidents may strike in a correlated fashion. Interconnected nodes [START_REF] Baer | Cyberinsurance in IT security management[END_REF][START_REF] Böhme | Modeling cyber-insurance: Towards a unifying framework[END_REF] and dominant products [START_REF] Baer | Cyberinsurance in IT security management[END_REF] are key causes for this interdependency. An incident in one organisation may thus cause or increase the likelihood of incidents in another organisation. This risk comes in addition to the potential impact of an incident on third parties, up and down in the supply chain [START_REF] Cashell | The economic impact of cyberattacks[END_REF]. For insurance companies, these characteristics increase risk of concurrent claims [START_REF] Baer | Cyberinsurance in IT security management[END_REF]. Additionally, cyber incidents may cause pay-outs on a number of different insurance policies [START_REF] Lloyd's | Cambridge Centre for Risk Studies: Business blackout -the insurance implications of a cyber attack on the us power grid[END_REF].

Table 1 gives a summary of key points from the literature related to these challenges.

Study of insurance companies

In addition to studying relevant literature, we have performed a study of experiences and practices of insurance companies when it comes to their cyber risk assessment of customers. The study reported in this paper is part of a bigger ongoing study with the overall goal of identifying what type of decision support and information is needed for evaluating cyber-insurance offerings, both from the perspectives of insurance companies and prospective customers.

Method

We have performed a study among key insurance companies offering cyberinsurance products in the Nordic market. The study included interviews performed in October/November 2015 and examination of relevant documentation. Relevant insurance companies for the study were identified by studying the web sites of such companies operating in the Nordic market, and the decision on which actors to approach for the study was based on the goal of covering the main actors in one country. The semi-structured interviews were carried out at the insurance companies' premises by one or two researchers, and the interviews were audio recorded and transcribed. The interview guide included the following topics: role of the interviewee; details of the cyber-insurance products offered; process for getting in touch with customers, evaluating risk and communicating policy terms to customers, and; future plans regarding cyber-insurance products. We talked with one representative from each company, and these were in underwriter or manager roles. Each interview lasted about one hour. All transcripts were thematically coded, and organised into thematic networks [START_REF] Attride-Stirling | Thematic networks: an analytic tool for qualitative research[END_REF]. As the Nordic cyber-insurance market is small with a limited number of providers, we do not give any further details about the insurance companies that participated, in order to preserve anonymity. However, we point out that the number of companies interviewed is small, but still comprise the main actors in one of the Nordic countries. Regarding the context of the study, it should be noted that cyber-insurance is a relatively new product in the Nordics, and it is only the last few years that pure cyber-insurance products have been marketed here. Many Nordic companies are still unaware of cyber-insurance products' existence.

Results

The insurance companies we have studied seem to have relatively similar approaches to differentiate insurance customers in terms of risk. In general, they seem confident that their current approach and evaluation is good enough. At the same time they experience challenges and constraints in evaluating prospective customers' cyber risk. Fig. 2 provides an overview of the central themes that came up in the interviews related to assessment of risk. In the following we go into these in more detail. An overview of the findings, related to the challenges identified in literature, can be found in Table 1.

Challenge Overview of state of the art Interview study

Understand risk factors and their interaction; no standard metric

• Uncertainty in what factors are most important for risk [START_REF]ENISA: Incentives and barriers of the cyber insurance market in europe[END_REF][START_REF] Marotta | A survey on cyberinsurance[END_REF][START_REF] Meland | Mitigating risk with cyberinsurance[END_REF][START_REF]Cybersecurity insurance workshop readout report[END_REF][START_REF] Toregas | Insurance for cyber attacks: The issue of setting premiums in context[END_REF].

• Challenging to achieve good measures of cyber risk [START_REF] Meland | Mitigating risk with cyberinsurance[END_REF][START_REF] Oppliger | Quantitative risk analysis in information security management: A modern fairy tale[END_REF].

• Insurers lack experience and standards [START_REF] Marotta | A survey on cyberinsurance[END_REF].

• Metrics considered by insurance companies are varied [START_REF] Baer | Cyberinsurance in IT security management[END_REF][START_REF]Cybersecurity insurance workshop readout report[END_REF][START_REF]Cyber risk culture roundtable readout report[END_REF].

• No clear priorities on what risk factors are most important.

• Do not have the competence in-house (use external security experts to perform assessments)

Access to data

• Lack of robust actuarial data is one reason for limited success of the cyber-insurance market [START_REF] Bandyopadhyay | Why IT managers don't go for cyber-insurance products[END_REF][START_REF]ENISA: Incentives and barriers of the cyber insurance market in europe[END_REF][START_REF] Ey | Mitigating cyber risk for insurers, part 2: Insights into cyber security and risk[END_REF][START_REF] Toregas | Insurance for cyber attacks: The issue of setting premiums in context[END_REF].

• Companies can be reluctant and unable to share reliable data on incidents [START_REF] Bandyopadhyay | Why IT managers don't go for cyber-insurance products[END_REF][START_REF] Gordon | A framework for using insurance for cyberrisk management[END_REF][START_REF] Marotta | A survey on cyberinsurance[END_REF][START_REF] Meland | Mitigating risk with cyberinsurance[END_REF][START_REF] Toregas | Insurance for cyber attacks: The issue of setting premiums in context[END_REF].

• Cyber-incident cost is not clearly defined and hard to measure [START_REF] Bandyopadhyay | Why IT managers don't go for cyber-insurance products[END_REF][START_REF] Cashell | The economic impact of cyberattacks[END_REF][START_REF] Marotta | A survey on cyberinsurance[END_REF].

• A competitive environment hinders information sharing among providers [START_REF]Cybersecurity insurance workshop readout report[END_REF][START_REF]Cyber risk culture roundtable readout report[END_REF].

• Challenges of information asymmetry [START_REF] Baer | Cyberinsurance in IT security management[END_REF][START_REF] Biener | Insurability of cyber risk: An empirical analysis[END_REF][START_REF] Böhme | Modeling cyber-insurance: Towards a unifying framework[END_REF][START_REF] Marotta | A survey on cyberinsurance[END_REF].

• Cyber-insurance is a new product, i.e. having limited historical data to build on.

• Getting hold of reliable data from customers can be challenging.

Demands on customers; efficient process for customers and insurance companies

• Clients may not like being dictated on how to mitigate cyber risk [START_REF] Marotta | A survey on cyberinsurance[END_REF][START_REF]Cyber insurance roundtable readout report -health care and cyber risk management: Cost/benefit approaches[END_REF], and may choose carriers with less stringent assessment practices [START_REF]Cybersecurity insurance workshop readout report[END_REF].

• Insurance companies aim to reduce effort of assessments [START_REF] Baer | Cyberinsurance in IT security management[END_REF][START_REF]Cybersecurity insurance workshop readout report[END_REF][START_REF]Cyber risk culture roundtable readout report[END_REF].

• Implementation of basic security measures is required, and risk assessment may introduce further requirements.

• Large customer base is needed, thus also efficient risk assessment processes.

• The sales process needs to be quick with a clear and easy-to-understand message.

Risk is constantly changing

• Rapid technological development and changes in attacker profiles may cause historical data to become outdated quickly [START_REF] Biener | Insurability of cyber risk: An empirical analysis[END_REF][START_REF] Marotta | A survey on cyberinsurance[END_REF][START_REF] Meland | Mitigating risk with cyberinsurance[END_REF][START_REF]Cybersecurity insurance workshop readout report[END_REF].

• It is challenging to stay updated in the field, things change quickly.

Globally correlated and interdependent risk

• Interdependent security and correlated risk; the security of a node is dependent on the security of other nodes and incidents may strike in a correlated fashion [START_REF] Baer | Cyberinsurance in IT security management[END_REF][START_REF] Böhme | Modeling cyber-insurance: Towards a unifying framework[END_REF][START_REF] Marotta | A survey on cyberinsurance[END_REF].

• Cyber incidents may cause pay-out on a number of different insurance policies [START_REF] Lloyd's | Cambridge Centre for Risk Studies: Business blackout -the insurance implications of a cyber attack on the us power grid[END_REF].

• When concerned about catastrophic incidents, this impacts policy terms (become more risk averse).

Table 1. Overview of identified challenges from literature and interviews ). They put some requirements on basic security measures that need to be in place for all customers, but they can also pose more specific requirements on individual customers. The criteria for deciding which security measures should be required from customers were not laid out by any of the interviewees. Some of the interviewees are concerned about what could be called catastrophic risk, i.e. the risk of incidents which impact the majority of their customers at the same time. The interviews however give no indication of factors being considered most important to understand and address such risk.

Constraints. An interviewee explained how they find it challenging to stay updated on the field of cyber security. This may in part be related to the fact that cyber-insurance is a new product where they have limited experience and actuarial statistics to build on. But it is also pointed out that cyber security is an area where things change quickly, depending on the creativity of various threat actors.

Access to data is one challenge often experienced. The information most commonly collected (such as revenue and business sector) is easily obtained and trusted. However, other types of information (data protection is particularly mentioned) may be more difficult to get hold of. The interviewees mentioned a number of reasons for this: competence available in the business on cyber security and privacy; availability of the right people that have the competence, and; unwillingness to share information. In addition comes the risk of getting erroneous answers to the questions asked related to risk.

Insurance companies need to have a large customer base for their cyberinsurance products. This means that they need to be able to handle a large number of customers in an efficient manner, but also that the sales process and the steps needed to become a customer is easy enough, not to hinder adoption. Additionally, the requirements regarding implementation of security measures must not be too strict. Currently, new customers are primarily reached through two channels (and this varies a bit between the insurance companies): insurance brokers and in-house sales personnel. Especially when in-house sales personnel are used, it seems important to exhibit a clear and easy-to-understand sales message and a quick process to sign up for cyber-insurance.

Making compromises. Today, insurance companies make several compromises in the process of evaluating risk, agreeing on policies and setting premiums. The following key areas came up in the interviews.

To get a large customer base, the insurance companies need to have efficient processes for evaluating risk and it must be easy for customers to buy insurance. At the same time, insurance companies need to be able to trust the assessments they make regarding cyber risk of customers, and thus need to get hold of a range of data related to risk. The collection process should ensure that data can be trusted. To achieve a balance between these needs, the insurance companies today differentiate their customers, and only perform thorough analysis of companies that are considered high risk. Differentiation is mainly done based on size, either of the company or of the amount insured. Small sized companies will have to answer only a very limited number of questions, ensuring the ability of the insurance company to reach and handle a mass of customers. For larger sized companies, a more thorough analysis is made, but also here some compromises have to be made to ensure an efficient collection process. Approaches made to data collection include questionnaires filled out by company representatives, interviews and security testing. Use of questionnaires requires less effort from the insurance company, but potentially at the cost of trust in the data. There is also the issue of how often to re-evaluate risk. The insurance companies are aware that cyber risk is dynamic over time. At the same time, they have limited capacity to re-evaluate risk on a regular basis.

The insurance companies in our study use third parties (cyber security experts) to perform the evaluation of company risks. Some interviewees state that a key reason for this is limited competence on cyber risk management in-house. The use of third party experts allows for high quality evaluations, and also opens possibility for learning from the third parties and thus gradually building more competence in-house. The insurance companies currently seem very dependent on the evaluations made by these third parties.

The insurance companies do put some baseline security requirements on their customers. As one interviewee states:"We don't want to insure a burning house!" At the same time, the companies want a volume of users, and to achieve this they cannot raise the bar too high for their customers. Currently, requirements on having anti-virus software and firewalls are common, either for becoming a customer or as part of the insurance terms. Insurance companies may have more strict requirements for larger customers, based on the results of the risk evaluation.

Approaches to efficient and thorough risk assessments

The results from our interview study supports the current literature on risk assessment challenges for cyber-insurance carriers. The interview subjects all represent insurance companies offering cyber-insurance in a market where this is a new and unknown product. They have limited historical data to build on, and there has not been many incident cases yet. This lack of experience and data and the need for an efficient sales process highly impacts their approach to risk assessment. In the following we explore two options for being able to perform a thorough, and yet efficient assessment of risk in this context.

Reusable sector-specific risk models

As part of the process of differentiating cyber risk, insurance companies collect information from customers in a structured and repeatable manner, for instance through reusable questionnaires. We do, however, get the impression that the process of determining the appropriate risk level based on the collected information is not always structured, and can be based on expert judgement alone, which may result in arbitrary and poorly documented decisions. A common alternative to expert judgement is to use a risk model. A risk model is often specified in a modeling language that provides a precise and well-documented way of determining/calculating the risk level based on the information in the model. Creating risk models, however, can be time consuming, and making a new model from scratch for every assessment may not be feasible for insurance companies. One way of mitigating this problem is to create a generic risk model which captures some common knowledge and which can be reused in each assessment. Furthermore, the reusable risk model may be used as a basis for determining what kind of information should be collected from the customers.

In many settings, building a reusable risk model may not be worth it because the risk models vary too much across different assessments. However, in the setting of differentiating cyber risk insurance customers, several factors suggest that the use of reusable risk models may be appropriate: [START_REF]Dependability management Part 3: Application guide Section 9: Risk analysis of technological systems Event Tree Analysis (ETA)[END_REF] The risk assessments have to be performed in a relatively short period of time, and the duration does not vary greatly from customer to customer. (2) The type of risks considered in each risk assessment are similar. These risks may for instance precisely correspond the type of cyber risks that are to be insured and which have precise legal definitions in the cyber-insurance terms and conditions, e.g. Data Breach, Material Interruption, or System Failure. (3) The information needed in each risk assessment is often collected from the customer in a structured manner and the type of information collected is similar across different customers.

In the field of risk assessment, several kinds of risk modeling techniques exists. These often build on tree-based and graph-based notations. Fault tree analysis (FTA) [19], event tree analysis (ETA) [START_REF]Dependability management Part 3: Application guide Section 9: Risk analysis of technological systems Event Tree Analysis (ETA)[END_REF] and attack trees [START_REF] Schneier | Attack trees[END_REF] are examples of the former and provide support for reasoning about the sources and consequences of unwanted incidents, as well as their likelihoods. Cause-consequence analysis (CCA) [START_REF] Nielsen | The cause/consequence diagram method as a basis for quantitative accident analysis[END_REF], CORAS [START_REF] Lund | Model-driven risk analysis: the CORAS approach[END_REF], and Bayesian networks [START_REF] Ben-Gal | Bayesian networks. Encyclopedia of statistics in quality and reliability[END_REF] are examples of graph-based notations.

None of the above mentioned techniques have dedicated support for risk model reusability. In our opinion, for a risk modeling technique to support reusability, it should at a minimum: [START_REF]Dependability management Part 3: Application guide Section 9: Risk analysis of technological systems Event Tree Analysis (ETA)[END_REF] Clearly distinguish between the information in the risk model which is parameterized (i.e. information that may vary across different risk model instances) and static information which should stay the same across different instances. ( 2) Provide clear guidelines for how the reusable risk model may be instantiated, i.e. how the parameterized information should be replaced with information specific to the instantiation domain.

Although none of the above mentioned risk modeling techniques have dedicated support for reusability, all of them can be extended to support this. One straightforward procedure for doing this in the setting of cyber risk insurance is as follows: (1) Make a risk model capturing general cyber attacks and cyberinsurance risks. (2) Identify and clearly mark which estimates in the risk model that are to be parameterized. ( 3) For each parameterized estimate, formulate a natural language question whose answer will allow the estimate to be determined for a given instantiation of the risk model.

By following the above steps, we obtain a set of questions in addition to a general risk model whose parametrized estimates are marked. To instantiate the risk model for a given cyber-insurance customer, we can give the questions to the customer, then determine the values of the parameterized estimates based on the answers to the questions, and finally determine the risk level using the guidelines of the risk modeling technique. As an example, consider the reusable risk model shown in Fig. 3 specified in the CORAS risk modeling language. A CORAS risk model is a graph whose nodes describe the occurrence of events, and whose edges describe causal rela-tionships between events represented by the nodes. Nodes may be annotated with likelihood values specifying how often events occur, and edges may be annotated with conditional likelihood values specifying the likelihood that one event leads to another event given that the former event has occurred. Consequences may also be specified. In CORAS, this is done by including special nodes called assets (the money bag in Fig. 3). The consequence that a node has on the asset may then be specified by drawing an edge from the node to the asset and labelling the edge with a consequence value. The meaning of the risk model specified in Fig. 3 is: Threat Hacker initiates the threat scenario Denial of Service (DoS) Attack with likelihood A1 which leads to the unwanted incident Material Interruption with conditional likelihood A2 which impacts the asset Availability with consequence A3.

The CORAS modeling language does not have explicit support for specifying parameterized values. However, in Fig. 3, we have indicated the parameterized values by the variables A1, A2, and A3. In addition, we have specified questions that may help estimate these values. In particular, question Q1 will help estimate the likelihood A1 of DoS attacks, question Q2 will help determine the conditional likelihood A2 that a DoS attack will cause Material Interruption given that the attack is initiated, and Q3 will help estimate the consequence A3 of the Material Interruption risk if it occurs.

In general, mapping an answer to an estimate may not be straightforward. However, in the current example, this is fairly straightforward since the questions have been identified based on the risk model (as opposed to building an appropriate risk model from a set of questions). To make matters even easier (for the insurance company), the answers to the questions could be constrained such that they can be directly replaced with the estimates they help determine in the risk model. For instance, the possible answers to question Q1 could be a choice between the likelihood values Low, Medium, or High, whose values have been given a precise meaning, e.g. Low could mean less than one time per ten years. Note that the likelihood estimate of the unwanted incident Material Interruption in Fig. 3 is not parameterized. However, this likelihood value can be calculated based on the other likelihood estimates in the diagram using the CORAS likelihood calculation rules. Using the CORAS calculation and verification rules will not be an issue, since we are only interested in using these rules on instances of the reusable model after the parametrized estimates have been replaced by actual values.

Although there may be clear advantages of using reusable risk models to differentiate cyber risk of insurance customers, more research is needed to determine how well this would work in practice. A particular concern is the feasibility of providing estimates that are sufficiently precise to determine an accurate risk level. If this proves to be infeasible, a possible solution is to only use qualitative estimates, or to have an explicit representation of uncertainty in the risk model if quantitative estimates are used. Another concern is how often the models need to be updated or the risk re-assessed, due to the dynamic nature of cyber threats.

A role for managed security service providers

Involving companies known as managed security service providers (MSSPs) can potentially reduce the footprint and improve the quality of cyber risk assessments for insurance companies. MSSPs are security companies who provide a range of security products and services, often including 24/7 monitoring, analysis and detection of malicious network traffic. In addition to having clear knowledge of implemented security controls at their customers' site, MSSPs notify their customers about incidents as they happen and sometimes they assist customers further with incident response services. At the MSSP the incidents are continuously being documented over time, through a range of incident categories, both aggregated and on a per-incident basis.

From a perspective where the insurance company wants to assess risk of the customer when calculating a premium, we believe an MSSP engaged by the customer can provide very specific input in an effective manner, compared to existing alternatives and practices. The MSSPs have already insight into their customer's security architecture, strengths and weaknesses, and in particular the historical information on incidents and how these are handled. Such involvement mitigates the challenge on access to data, as identified in Table 1. In addition, MSSPs need to stay updated on general security and threat research, developing knowledge on the overall threat landscape including both public information as well as threat intelligence collected through monitoring their other customers, and through threat intelligence sharing channels. A partnership with an MSSP could help an insurance company better understand cyber risk, make the assessment process more efficient using previously collected knowledge, and staying up-to-date in terms of risk for the particular customer, general trends and globally correlated risks which may accumulate to catastrophic incidents.

For MSSPs and insurance companies to collaborate, incentives are needed for the customer organisation to allow information sharing between them. The MSSP holds very sensitive and detailed data about their customers, although some of this information can be made less sensitive through e.g. aggregation, anonymization or generalisation. Since MSSPs have their own security and risk experts in-house, the process of creating a risk map of their customer could for instance be offered as a service to the insurance company, without disclosing all of the underlying details. The result would still provide a reliable -and much richer -basis for the insurance company to assess risk, than simply relying on a self-evaluation by the customer.

While this approach limits the need for a potential insurance customer to expose sensitive information any further than already done, it also reduces their own workload related to obtaining insurance offers/coverage. It could also increase the likelihood that MSSP customers obtain a lower insurance premium as a result of implemented security measures, due to larger transparency for the insurance company and documented controls in form of the implemented security services by the MSSP. At the same time, selling this particular service provides an incentive for MSSPs to get involved in the value-chain, along with the potential upside for MSSPs on selling incident response services through insurance companies whenever this serves as coverage. Finally, this process reduces costs for the insurance company since it would be less cost-demanding than employing another third party assessment from someone without previous knowledge of the potential customer's security architecture, controls and incident history.

If the MSSP industry were able to agree with the insurance companies on a standard for disclosing risk information about their customers, we believe a win-win situation could arise for all parties involved. In practice this is a matter of agreeing on a level of detail and a format of the information to be shared between the MSSPs and the insurance companies, and a portofolio of products they can collaborate on. It is an open question to insurance companies whether more transparency should automatically incur lower premiums. There could also be implemented mechanisms where the MSSP automatically alerts the insurance company directly about incidents at the insured company in a timely and standardised manner, making it easier for the insurance company to provide the right assistance in less time, and at an earlier stage, which may help in limiting costs of incident handling. One could also imagine that the MSSP would be directly authorised by the insurance company in these situations to implement mitigations within agreed coverage limits, which would likely be more effective than using a third party without a day-to-day relationship with the insured company.

Discussion

In this paper we have presented practice and challenges regarding assessing cyber risk of cyber-insurance customers, from the viewpoint of insurance companies. Practice and challenges have been identified by studying literature, and through a study of insurance companies in the Nordic market. It needs to be noted that the study of insurance companies reported in this paper only include a few companies, due to the size of the Nordic market, and in order to maintain their anonymity we do not provide any specific details on these companies. As a consequence of these limitations, we do not claim that our results are generalizable to all insurance companies. However, we believe the validity of our results is strengthened when considered together with known challenges identified in literature.

Table 1 compares the main results from the literature study with our study of insurance companies. The results are very much in line; the same challenges that are described in literature are experienced by the insurance companies we have studied. However, what is more of a concern for the companies that we have studied, compared to the emphasis it has been given in the literature, is the need for an easy to understand product and an efficient sales process. As already mentioned, the Nordic market for cyber-insurance is characterised by this being a new and unknown product. This impacts the approach taken to risk assessments. In addition to the need for an efficient sales process, the insurance companies have limited experience with the product and little historical data to build on, though they can to some extent use experiences from cyber-insurance in other markets.

To improve cyber risk insurers' ability to have efficient, yet thorough analysis of cyber risk we have proposed two ways forward: reusable sector specific risk-models and including MSSPs in the value chain. These are not completely new ideas; the potential for developing a generic modeling structure for use with cyber-insurance has been discussed in a roundtable event organised by Department of Homeland Security [29] as part of their work with building a repository for sharing incident data [START_REF]Enhancing resilience through cyber incident data sharing and analysis: The value proposition for a cyber incident data repository[END_REF], and in these discussions the potential role of security companies came up regularly when it comes to providing incident information [START_REF]Cybersecurity insurance workshop readout report[END_REF][START_REF]Cyber risk culture roundtable readout report[END_REF][START_REF]Cyber insurance roundtable readout report -health care and cyber risk management: Cost/benefit approaches[END_REF][29]. The study we have performed further underlies the need for improved approaches, and in this paper we have provided more details on how one could go about building such re-usable models, and strengthen the input to the models by including MSSPs in the risk assessment process.

There is a need for more research in order to better understand how cyber risk assessments could be improved related to cyber-insurance. We have only studied this topic from the viewpoint of insurance companies, but other actors also have important roles in this respect; customers, brokers and third-party risk assessors. Further, the approaches we have outlined that can potentially improve current practice need to be evaluated to assess whether or not they will have the desired effect.

Conclusion

Insurance companies need to be able to assess cyber risk of prospective cyberinsurance customers. Many challenges related to performing such analysis have been identified in literature, and our study of insurance companies offering cyberinsurance to the Nordic market shows that these challenges described in the literature are relevant for insurers that operate in a market where cyber-insurance is a new and relatively unknown product. In such a market, the need for an efficient sales process and a clear sales message further constrains how insurance companies perform assessments of cyber risk. In this paper we have proposed two possible approaches to improve insurers' ability to perform efficient, yet thorough analysis of cyber risk: building reusable sector-specific risk models and collaborating with MSSPs.

Fig. 1 .

 1 Fig. 1. The risk assessment process and challenges

Fig. 2 .

 2 Fig. 2. Thematic network showing how insurance companies differentiate between potential customers.

Fig. 3 .

 3 Fig. 3. Example of a reusable risk model specified in CORAS

Acknowledgments. This research has been performed as part of the inSecurance project, which is a strategic and internally funded research project at SINTEF ICT. We would like to thank the representatives from the insurance companies that participated in the interviews for sharing their experiences with us. We would also like to thank our colleagues Per Håkon Meland and Aida Omerovic for taking part in discussions leading up to this paper and for commenting on the interview guide, and our colleague Bjørnar Solhaug for performing and transcribing the interviews.