Simona Cavallini

Cristina D'

Margherita Volpe

Stefano Armenia

Camillo Carlini

Elisabeth Brein

Pierluigi Assogna

A SYSTEM DYNAMICS FRAMEWORK FOR MODELING CRITICAL INFRASTRUCTURE RESILIENCE

Keywords: Interdependencies, critical events, domino effects, system dynamics

In recent years, awareness of the potential consequences associated with a major disruption to the critical infrastructure has grown among public and private entities. Indeed, traditional and emerging threats endanger service continuity and, by extension, the normal functioning of modern society. This paper presents an approach for modeling the effects of critical infrastructure failures as a result of unexpected events. The transportation, energy and telecommunications infrastructures are modeled using a system dynamics approach. The work constitutes a component of the CRISADMIN Project that is focused on developing a tool to evaluate the impacts of critical events. The ultimate objective of the project is to provide decision makers with a sophisticated tool to help them mitigate negative effects in emergency situations. The prototype tool described in this paper leverages case studies of terrorist attacks and floods that have occurred in Europe.

Introduction

Critical infrastructures are the backbone of modern society, enabling the vital functionalities that support economic and social interactions. The European Commission's 2008/114/EC Directive [START_REF]Identification and Designation of European Critical Infrastructures and the Assessment of the Need to Improve Their Protection[END_REF] defines critical infrastructure as "an asset, system or part thereof located in Member States which is essential for the maintenance of vital societal functions, health, safety, security, economic or social well-being of people, and the disruption or destruction of which would have a significant impact in a Member State as a result of the failure to maintain those functions." It is important to note that system failures in a specific critical infrastructure sector can, due to their strategic role in the socio-economic context, produce domino effects that can potentially impact all aspects of society. Understanding the effects and strategic interconnections are essential when responding to events, setting policies and determining protective investments.

Thurlby and Warren [START_REF] Thurlby | Understanding and managing the threat of disruptive events to the critical national infrastructure[END_REF] state that, in order to rank preventative measures, the economic costs and potential savings (i.e., reduced casualties and/or economic losses) must be evaluated. Thus, there is a growing need to understand the costs for society as a whole -beyond those of the initially-impacted infrastructures -to fully comprehend the magnitude of an event and make appropriate response decisions.

A number of powerful simulation tools have been developed to help understand how networks may be affected by major incidents, many of which help organizations to improve their response readiness. Nevertheless, the relationship between long-term strategic choices and the ability of infrastructure networks to withstand disruptive events are not well understood. Indeed, decision making concerning investments in critical infrastructure assets, particularly those related to network control systems and the people who manage the systems, have not been thoroughly investigated to determine the long-term implications. While it is clear that spending less on assets, systems and people will degrade a system, it is not obvious how much impact any particular choice has over an extended period of time. The primary issues that need to be addressed are: How long-term choices related to strategic issues make a network more resilient.

How these choices and others can minimize service loss when disruptive events occur.

How strategic and operational choices can minimize the time taken for a network to recover and, thus, minimize the total cumulative loss of services.

The Critical Infrastructure Simulation of Advanced Models for Interconnected Network Resilience (CRISADMIN) Project studies the effects produced by critical events in an environment in which the interdependencies among several critical infrastructure sectors are modeled using a system dynamics approach and simulated in a synthetic environment. This paper discusses the key features of the methodology. The intention is to provide insights into the activities and expected outputs of the project, providing researchers and professionals with a methodology for crisis management.

CRISADMIN Approach

The CRISADMIN Project is focused on developing a tool for evaluating the impacts of critical events on critical infrastructures. The tool is intended to serve as a decision support system that is able to test and analyze critical infrastructure interdependencies, determine the modalities through which they are affected by predictable and unpredictable events (e.g., terrorist attacks and natural disasters), and investigate the impacts of possible countermeasures and prevention policies.

To achieve these challenging objectives, a three step approach has been formulated:

Theoretical Model Definition: The first step is to define the system characteristics in order to establish the investigative boundaries and key reference points. This objective is achieved through the formulation of a theoretical model that identifies variables and parameters that best represent (or approximate) the infrastructures of interest. Special attention is focused on the identification of social system variables (i.e., "soft" parameters that are particularly difficult to quantify). Through careful analysis of the literature, these variables are represented in a manner compatible with system dynamics.

System Dynamics Model Development: Causal relations between the parameters defined in the theoretical model are identified; this facilitates the construction of a number of causal maps. The causal maps provide the foundation for the simulation model structure that is validated using real case studies.

Data Collection: Quantitative data concerning critical infrastructure functionality is collected from a number of case studies. In addition, data related to the socio-economic framework is gathered according to its availability and reliability with reference to critical events that have occurred in Europe in recent years.

Starting with the definition of a theoretical reference framework, the goal is to design a system dynamics model that constitutes the logical base for developing the decision support system. The effort engages case studies for model development and analysis. The models are integrated within the decision support structure to produce a readily accessible and usable decision making tool.

Theoretical Model

The theoretical model defines the main factors that should be considered in an emergency situation. The goal is to enhance the preparedness and response capability of all the involved actors in order to mitigate and recover from the negative effects of a catastrophic event. The main factors are investigated in terms of mutual influences, those that reinforce and those that dampen the effects of an event. Special attention is focused on the involved actors (i.e., victims, spectators and individuals responsible for managing the emergency) [START_REF] Bourque | Survey research[END_REF].

As in all complex environments, the vast majority of factors in emergency situations are highly interconnected. The primary objective of the theoretical model is to identify the main dependencies that impact the evolution of an event. Territorial features, the socio-economic environment, event timing (e.g., time and duration) and actor preparedness are included in the analysis. In the CRISADMIN Project, the effects of a critical event are studied in the context of three critical infrastructure sectors, namely transportation (private and public), energy (electricity distribution and consumption) and telecommunications (mobile and fixed).

Data domains are grouped according to the parameters included in the model. Specifically, four data domains are considered:

Territory: This domain includes the set of variables and parameters that describe the geographic features of the territory. Territorial characteristics are particularly relevant to natural disasters; however, they may also affect the efficiency of responses in other critical situations (e.g., high territorial diversity exerts a negative influence on the promptness of emergency transportation). In this data domain, the main elements are the territorial factors and geographical nature (e.g., extension and locality) that impact vital services and social aspects.

Environment: This domain refers to the set of variables and parameters related to the presence and activities of human beings in the territory, such as energy-related supply chain capacity, public transportation, population density and socio-economic patterns in the affected area. In the case of human-initiated critical events, environmental parameters are essential to successful crisis response.

Apparatus: This domain includes the set of variables and parameters related to the professionals and operators who manage the effects of catastrophic events and the subsequent recovery. Typically, the apparatus includes multiple agencies and organizations, each of which have a specific role in managing minor emergencies as well as unexpected critical events. In some countries, civil authorities coordinate the activities of all the various apparatus organizations in order to mitigate the effects of a critical event.

Events: This domain refers to the set of variables and parameters that define "normal" conditions. The data describes the evolution of normal situations over time (in contrast, the geographical features in the territory domain are time independent). Data related to the environment and apparatus depend on the normal life-cycles and are tied to the hour of the day (e.g., work hours and commuting hours), day of the week (e.g., workday, weekend, bank holiday and special days) and month of the year (e.g., festivals and vacation periods). These dependencies, which can be more or less substantial for the different variables, are considered when modeling the evolution of a critical event from the very first moments after it occurs. After the first parameter adjustment at t 0 , the evolution of an event is generally considered to be independent of the hour, day and month because of the emergency effects. sectors are shown to illustrate the items that require investigation when responding to a critical event.

Identification of Social System Variables

The data domains capture the stage at which a critical event occurs, where countermeasures should occur and how the damage should be assessed. For this reason, social aspects involved in the preparedness and in the reaction to critical situations are included in the environment and apparatus domains. Special attention should be focused on the actors that participate in the activities being modeled and on the ordinary events that represent normal conditions.

Frequently, discussions about the effectiveness of crisis management focus on the material side (e.g., engineering and structural solutions, effective information technology and transportation networks, safeguarding electric power networks from overload and increasing the number of first responders). These discussions often ignore the thoughts, attitudes, expectations and behavior of individuals and groups who are affected by the crisis or are involved in their management. Indeed, disregarding the social and psychological aspects is problematic because each infrastructure, despite its material nature, is always embedded in the social environment. As Orlikowski and Scott [START_REF] Orlikowski | Sociomateriality: Challenging the separation of technology, work and organization[END_REF] argue, technology is always technology in practice, highlighting the fact that the same technology will be used in very different ways depending on the social context in which it is used; this fact is also true in crises. Therefore, an important premise of the CRISADMIN Project is that, in order to have effective crisis management, it is of vital importance not only to understand how infrastructures and technologies work, but also to understand "how relations and boundaries between humans and technologies are not given or fixed, but enacted in practice" [START_REF] Orlikowski | Sociomateriality: Challenging the separation of technology, work and organization[END_REF]. The realistic modeling of crises clearly requires the inclusion of social variables.

When considering the social variables to be incorporated, it is important differentiate between two categories of human actors. One category includes the people who are actively effected by the crisis. The other category comprises the individuals who attempt to manage the crisis and the subsequent recovery. A critical event induces behavioral changes in both categories of people. Note, however, that these two categories are not necessarily mutually exclusive.

The literature review undertaken by the CRISADMIN Project focused on human behavior in social systems during the response phase. Emphasis was placed on the impact on the individuals affected by the crisis as well as on the individuals involved in managing the crisis. Possible interactions, including inter-organizational coordination in emergency response, leadership in crisis situations and approaches for communication and information dissemination, were taken into account.

The literature review was by no means limited to a specific type of critical event. Indeed, the fundamental assumption was that social system variables in crisis responses are generic in nature and applicable to disparate crises. The CRISADMIN Project specifically considered papers related to crisis management from the theoretical and empirical points of view, papers related to psychological and organizational knowledge based on empirical analysis, and papers related to psychological and organizational knowledge dealing with noncrisis management and based on empirical research. The review identified a total of 34 social variables.

The literature review also yielded several general observations related to the importance of social system variables when modeling critical events: For effective crisis management, material needs and social needs should be considered simultaneously.

Adequate communication is essential immediately after a crisis occurrence.

Communication flows are core aspects of strategies for systematic crisis management.

The need for a communication strategy has to be embraced by first responders to improve crisis management.

Information sharing is significant to successful inter-organizational cooperation.

A longitudinal perspective should be considered; experience from past critical situations affects current crisis response and reactions.

In addition, cultural and societal settings (e.g., values, attitudes and demographics) that strongly influence the preparation for and the reaction to critical events with regard to victims and first responders in specific environments were taken into account.

System Dynamics Methodology

The CRISADMIN Project employs a computer simulation modeling methodology based on systems dynamics and feedback for studying and managing complex issues and problems encountered during crisis events. The feedback systems, such as social response, are defined as a collection of interacting elements working together for a certain purpose. The key element is to consider the concatenation of causal relations through which any component of a system can influence the behavior of other components that may be proximal or distant in terms of the apparent connections [START_REF] Friedman | Learning to make more effective decisions: Changing beliefs as a prelude to action[END_REF].

Originally developed in the 1950s to help corporate managers improve their understanding of industrial processes, system dynamics is currently used to understand the dynamic behavior of complex systems [START_REF] Friedman | Learning to make more effective decisions: Changing beliefs as a prelude to action[END_REF]. The application is based on the fact that the structure of any system relies on circular, interlocking and time-delayed relationships among its components.

Sterman [START_REF] Sterman | Business Dynamics: Systems Thinking and Modeling for a Complex World[END_REF] stated that the main properties of a system that can be successfully represented using a system dynamics approach are: (i) presence of quantities that vary over time; (ii) variability based on causal dependencies; and (iii) feedback loops containing the main causal influences of a closed system. Additionally, Sterman argued that system dynamics, as a decision modeling approach, is very applicable in contexts where standard analysis is made difficult by the wide range of available data. It is particularly applicable to systems that are highly influenced by soft variables, which are not directly measurable (e.g., trust in first responders, attitudes of the public and panic diffusion).

In recent years, the system dynamics approach has been used to prevent and manage security/defense issues, primarily because it takes into account randomness and interdependencies that characterize behavior in real-world environments. This is made possible by including the soft variables typical of interrelated social systems. The idea behind the system dynamics approach is that if the system structure defines the behavior of the system, then accurately analyzing and determining the interrelationships among the various components of the system produces an accurate understanding of the dynamics of the system [START_REF] Sterman | Business Dynamics: Systems Thinking and Modeling for a Complex World[END_REF].

The CRISADMIN Project uses system dynamics to forecast the evolution of the modeled components (i.e., territorial features, timing of the critical event, environmental factors, types of actors involved and social behaviors) from the occurrence of a critical event until the realization of the subsequent impacts. The holistic approach of system dynamics requires that the entire context be considered and that factors perceived as weak or not strictly related be disregarded. This aspect is intended to help avoid defining a model that is difficult to manage and/or interpret. The identification of relevant influences within the system dynamics framework makes it possible to understand the connections among critical infrastructures and to model the impacts of critical events, taking into account the dynamics of an infrastructure as a function of the operations of other critical infrastructures that are not affected directly. System dynamics simulations must represent the main mutual influences of the parameters identified in the theoretical model, defining each influence as positive (i.e., reinforcing) or negative (i.e., dampening), and the related value and timing. Table 1 presents example influences among the identified variables.

The overall model uses interactions among influences and additional information to estimate the total direct and indirect impacts arising from a critical event. This approach allows for the comparison of impacts within the socioeconomic context that is represented as a dynamic system. Once they are consolidated, the proposed influences are tested using data gathered from relevant case studies.

Data Collection

In order to apply the CRISADMIN approach, four critical events related to previous terrorist attacks and floods were identified and analyzed. The following criteria drove the selection of the events: Threat Likelihood: The frequency of terrorist attacks has increased since the events of September 11, 2001, reinforcing the view that a terrorist attack is a real possibility [START_REF] Enders | The Political Economy of Terrorism[END_REF]. Floods are also becoming more frequent.

Historical Event Frequency: Special attention is placed on terrorist attacks in Spain and the United Kingdom -countries that have suffered from ethnic terrorism for decades, from the Euskadi Ta Askatasuna (ETA) and the Irish Republican Army (IRA), respectively, and, more recently, have had to deal with attacks by Islamic terrorist groups. Critical Infrastructure Impact: Terrorist attacks and floods destroy essential assets, and impact critical infrastructures directly or indirectly. These events tend to have major impacts on the transportation, energy and telecommunications sectors.

Madrid Bombings (2004)

On the morning of March 11, 2004, explosive devices were detonated aboard four commuter trains in Madrid [START_REF] Owen | Bomb squad link in Spanish blasts[END_REF]. The affected trains were traveling on the same line and in the same direction between the Alcala de Henares and Atocha stations. A total of thirteen improvised explosive devices were placed on the trains. Ten of the devices exploded; two of the remaining three devices were detonated by Spanish Police bomb disposal experts at the Atocha and El Pozo stations. The thirteenth bomb was not found until later in the evening, having been stored inadvertently with luggage taken from one of the trains. In the following days, official investigations made by the Spanish Judiciary determined that the attacks were directed by a Muslim terrorist cell, which was inspired by al-Qaeda, although no direct al-Qaeda participation was ever established [START_REF] Nash | Madrid bombers "were inspired by Bin Laden address[END_REF].

The terrorists boarded the four commuter trains, each with a capacity of 6,000 passengers. They hid thirteen bomb bags (backpacks) amongst passenger luggage in several carriages before disembarking. Each backpack contained approximately ten kilograms of dynamite; some of the bags were filled with nails and other shrapnel to cause serious wounds to commuters. The explosive devices, which were activated by mobile phone alarms, were set to explode at various commuter stations to maximize casualties and property damage.

The bombings killed 177 people instantly and wounded approximately 1,858 others. Fourteen of the injured people subsequently died, bringing the final death toll to 191. More than 550 staff members and 100 vehicles from SAMUR Civil Protection were involved in the rescue and management activities. Within 90 minutes, SAMUR mobilized more than 325 people, increasing their staffing from 75 to 400 people, and recalled 70 vehicles. Healthcare-related activities in the emergency areas were performed by SAMUR and other local institutions.

The transportation sector was the only one to be directly affected by the bombings, in particular the four trains and the stations where the explosions occurred. The four trains were on the same track, heading towards Atocha Station (main commuting point in Madrid), El Pozo Station, Santa Eugenia Station and Calle Tellez Station. The energy and telecommunications infra-structures were not directly targeted by the bombing attacks. However, the telecommunications infrastructure experienced massive overloads due to general panic and crisis management needs.

London Bombings (2005)

On July 7, 2005, three London subway stations (Aldgate, Edgware Road and Russel Square) were attacked by suicide bombers. In addition, a bomb was placed in a double-decker bus that detonated in Travistock Square [START_REF]London Assembly[END_REF]. The bombings were carried out by four Islamic extremists, who were angered by Britain's involvement in the Iraq War. At about 8:50 A.M., three almost simultaneous explosions detonated in the tunnel between Liverpool Street and Aldgate stations, on the line at Edgware Road and in a Piccadilly Line tunnel between King's Cross and Russell Square.

Almost an hour later, at 9:47 A.M., the bomb placed in the double-decker bus was detonated at Travistock Square. The location of the bomb inside the bus resulted in the front of the vehicle remaining mostly intact. Indeed, most of the passengers in the front of the top deck survived, as did those near the front of the lower deck, including the driver. Individuals at the top and lower rear of the bus suffered more serious injuries. Several passersby were injured by the explosion and some surrounding buildings were damaged by debris. In order to ensure the maintenance of normal security and civil protection services in the city, the choice was made to send only critical staff to the bombing sites -leaving non-essential personnel, equipment and materials at headquarters in the stand-by state. Only the transportation infrastructure was directly affected by the bomb blasts.

Central and Eastern Europe Floods (2002)

In August 2002, severe flooding affected portions of Austria, the Czech Republic and Germany [START_REF] Ulbrich | The Central European floods of August 2002: Part 1 -Rainfall periods and flood development[END_REF]. Heavy rainfall from storms that crossed central Europe during early August triggered sequential flood waves along two major river systems. The flood waves moved down the Danube through Austria and down the Vltava and Elbe rivers in the Czech Republic and Germany. The flooding event covered a period of approximately fourteen days from August 6 until August 20, 2002. The event included precipitation as well as flash floods along the involved rivers in Central and Eastern Europe.

The August 2002 floods were due to two major factors: unusual meteorological conditions and human activities (e.g., housing construction, land drainage and deforestation). The flood event was triggered by unusual meteorological conditions, which included two periods of intense rainfall during the first half of August 2002. As usual, the water temperatures in the Adriatic and Mediterranean were significantly higher in August than in the spring, causing substantial amounts of atmospheric moisture that fueled the extreme rainfall. The first period of rain on August 6 and 7, 2002 fell in the southwestern Czech Republic and northeastern Austria, immediately north of a weak area of low pressure. Rainfall accumulations were generally less than 125 mm over the two-day period, but intense rainfall of up to 255 mm was observed in some locations.

The rainfall triggered flood waves in the upper portions of the Danube and Vltava catchment areas. One flood wave progressed down the Danube through Austria, Slovakia and Hungary, causing minor damage. A more critical flood wave progressed down the Vltava through Prague and down the Elbe through northern Bohemia and Germany. Upon reaching Germany, the flood waters in the Elbe inundated Dresden, causing damage to residential and commercial property as well as many historical buildings in the city center. The increase in river height in Dresden was more gradual and of greater magnitude than the flood peak in Prague. Although Prague itself was hardly hit by the flash flood, damage occurred in the historical and residential parts of the city center.

The greatest number of fatalities (58) was caused by floods resulting from the first wave on the eastern coast of the Black Sea. Seventeen people died in the Czech Republic, 21 in Dresden and more than 100 fatalities were reported across Europe. Direct and indirect impacts on the transportation and energy infrastructures were registered.

6.4

United Kingdom Floods (2007)

In June and July 2007, the United Kingdom was stricken by a series of severe floods arising from heavy rainfall during an unseasonably wet weather pattern [START_REF]The summer floods: What happened[END_REF]. The severe flooding events were attributed to two major causes: (i) position of the Polar Front Jet Stream; and (ii) high North Atlantic sea surface temperatures.

Heavy rainfall is not unusual in the United Kingdom during the summer months, but the frequency and spatial extent of the rainfall in June and July 2007 were unprecedented. Exceptional rainfall events occurred on June 25 and July 20, which caused widespread floods across England. The floods ranged from small, localized flash floods to widespread events affecting major river basins. First, northeastern England was badly affected by severe rainfall events in June, which caused floods in Sheffield, Doncaster, Rotherham, Louth and Kingston-upon-Hull. Some areas were hit again by further flooding after severe rain in July that affected a much larger area of central England, including Oxford, Gloucester, Tewkesbury, Evesham and Abingdon. The intense rainfall saturated the catchment areas, resulting in rivers flooding their banks in several major river basins. Disruptions to power and water supplies during the July floods were caused by flooding at the Castlemeads power substation near Gloucester and at the Mythe water treatment plant in Tewkesbury.

A total of thirteen people died as a result of the floods and approximately 48,000 homes were damaged. The scale and speed of the floods came as a shock. Although most people were aware of the impending heavy rain that was forecasted, they did not anticipate the magnitude of the rainfall. Indeed, most people involved in the incident had never experienced such flooding and did not know how to react. At the peak of the flooding, around 350,000 homes across Gloucestershire were left without water and 50,000 homes without power.

CRISADMIN Prototype

The CRISADMIN Project seeks to demonstrate, by means of a prototype, that a flexible system dynamics modeling engine can assist first responders and decision makers in managing critical events. During actual events, knowledge of the past, coupled with the current aspects of a given context, form the basis for selecting modeling parameters and defining influences.

The CRISADMIN decision support system takes into account experience gained through participation in projects associated with the design of modeling methods and tools for monitoring and contrasting emergencies [START_REF] Assogna | Critical infrastructures as complex systems: A multi-level protection architecture[END_REF]. The decision support system incorporates a three-tiered architecture: (i) a back-end that stores variables and parameters associated with the four domains; (ii) a core that houses the system dynamics modeling engine; and (iii) a front-end that maintains the parameters, activates the functions and presents results.

The simulation model will be made available to institutions and organizations across the European Union -public entities (e.g., civil protection and fire brigades) as well as private entities (e.g., infrastructure asset owners and operators). Crisis management is typically performed in interconnected operations control rooms (OCRs) that continuously monitor critical events. The CRISADMIN decision support system is designed for use by analysts in OCRs as they coordinate activities during critical events. The decision support tool will be used to support operational decisions that benefit from the continuous monitoring capabilities provided by OCRs. The tool will provide decision makers with a starting point that is both expandable and customizable. The tool environment will also engage several fixed and non-customizable scenarios and situations that encompass different crisis situations. This feature will enable decision makers to understand the dynamics of interacting critical infrastructure assets. The prototype will also provide decision makers with points of reference as they select appropriate policy alternatives for crisis management.

Conclusions

Decision makers responsible for infrastructure protection and crisis management must understand the consequences of policy and investment options before they enact solutions. This notion is particularly important due to the highly complex alternatives that must be considered when protecting critical infrastructures in the current threat environment. An effective way to examine and pursue trade-offs involving risk reduction and protection investments is to utilize a decision support system that incorporates information about threats and the consequences of disruptions. System dynamics modeling, simulation and analysis can be used to conduct impact assessments and risk analyses based on realistic scenarios.

The system dynamics approach developed under the CRISADMIN Project provides decision makers with a methodology for understanding and evaluating potential risks. The approach can be readily applied in contexts where standard analysis is made difficult by the wide range of available data and/or relationships. The approach is especially suited to systems that are greatly influenced by the "soft" variables associated with human behavior.

The CRISADMIN effort has identified the main parameters associated with the dependencies that impact the evolution of critical events. The result is a simple, yet effective, representation of how an event influences the behavior of a larger interconnected system. As new threats from terrorism and environmental factors emerge, a tool that enables decision makers to anticipate the impacts of critical events would provide them with precious insights for crafting protection strategies and implementing response actions.

Figure 1

 1 Figure 1 presents the CRISADMIN theoretical model with the four data domains. Examples of parameters related to the three critical infrastructure

Figure 1 .

 1 Figure 1. CRISADMIN theoretical model.

  Meanwhile, Europe has seen increased flooding incidents; examples are the 2002 Glasgow flood in the United Kingdom, the 2001 Po river floods in Italy and the 2011 Genoa flood in Italy.

Four

  case studies were used to apply and validate the CRISADMIN approach. The case studies include: (i) Madrid bombings of 2004; (ii) London bombings of 2005; (iii) Central and Eastern Europe floods of 2002; and (iv) United Kingdom floods of 2007. The following sections briefly describe the selected case studies and highlight their essential elements and impacts.

Table 1 .

 1 Example influences among the identified variables.

	Influencing Parameter Influenced Parameter Notes
	Crest (Flood)	(+) Inundation area	Calculated using
			elevation map
	Total Inundation Area	(+) Involved structures	Ascertained by
			first responders
	Energy Production	(-) Electricity disruption Adjustment of
	Damage		rate on lost power
	Electricity Station	(-) Electricity disruption Adjustment of
	Damage		rate on lost power

Acknowledgements

This research, conducted by personnel from the Department of Computer, Control and Management Engineering of La Sapienza University, FORMIT Foundation, Erasmus University Rotterdam, Theorematica and Euro Works Consulting, was performed under the CRISADMIN Project. The CRISAD-MIN Project is supported by the Prevention, Preparedness and Consequence Management of Terrorism and Other Security-Related Risks Program launched by the Directorate-General of Home Affairs of the European Commission.


Mohammed Talat Khouj

Sarbjit Sarkaria

Cesar Lopez

Jose Marti

REINFORCEMENT LEARNING USING MONTE CARLO POLICY ESTIMATION FOR DISASTER MITIGATION

Keywords: Disaster response, Monte Carlo estimation, decision assistance agent

Urban communities rely heavily on the system of interconnected critical infrastructures. The interdependencies in these complex systems give rise to vulnerabilities that must be considered in disaster mitigation planning. Only then will it be possible to address and mitigate major critical infrastructure disruptions in a timely manner.

This paper describes an intelligent decision making system that optimizes the allocation of resources following an infrastructure disruption. The novelty of the approach arises from the application of Monte Carlo estimation for policy evaluation in reinforcement learning to draw on experiential knowledge gained from a massive number of simulations. This method enables a learning agent to explore and exploit the available trajectories, which lead to an optimum goal in a reasonable amount of time. The specific goal of the case study described in this paper is to maximize the number of patients discharged from two hospitals in the aftermath of an infrastructure disruption by intelligently utilizing the available resources. The results demonstrate that a learning agent, through interactions with an environment of simulated catastrophic scenarios, is capable of making informed decisions in a timely manner.

Introduction

All of modern society, but in particular urban communities, rely heavily on the system of interconnected critical infrastructures. These systems are inherently complex in terms of interconnections and interdependencies. Thus, they are vulnerable to major disruptions that could cascade to other dependent systems with possible disastrous consequences. For example, the Indian Blackout of 2012 -the largest power blackout in history -caused massive disruptions to medical facilities, transportation systems, water treatment plants and other interconnected infrastructures. It resulted in the loss of power to 600 million people, trapping miners, stranding railway passengers and plunging hospitals into darkness [START_REF] Daniel | India power cut hits millions, among world's worst outages[END_REF]. Such catastrophic incidents reveal the need for efficient planning and, more importantly, the need for careful decisions to be taken during the first few hours following an incident. The decisions are critical to successful mitigation, damage management, death prevention, injury, structural loss, control of financial costs and, ultimately, the overall resolution of the crisis [START_REF] Kowalski-Trakofler | Judgment and decision making under stress: An overview for emergency managers[END_REF].

This paper describes an intelligent decision making system that optimizes the allocation of resources following an infrastructure disruption and suggests how the resources may be utilized during disaster response. An underlying intelligent learning agent interacts continuously with a simulated environment and uses reinforcement learning (RL) to discover a policy that optimizes a longterm reward. The learning system employs Monte Carlo (MC) estimation for policy evaluation in reinforcement learning to gain experiential knowledge over a massive number of simulations using the interdependent critical infrastructure simulator (i2Sim). The approach enables the learning agent to explore and exploit the possible trajectories that lead to an optimum goal in a reasonable period of time.

Related Work

This section describes related work in the areas of disaster mitigation in interdependent critical infrastructures, agent-based modeling for disaster mitigation and disaster mitigation applications using reinforcement learning.

Disaster Mitigation

Critical infrastructures are characterized by complex interconnections and interdependencies. These systems are vulnerable to major disturbances that can cascade to other dependent systems, potentially leading to national disasters (Figure 1). Interdependencies between infrastructures are bi-directional relationships through which the state of one infrastructure is influenced by or correlated with the states of other infrastructures [START_REF] Rinaldi | Modeling and simulating critical infrastructure and their interdependencies[END_REF]. Thus, it is essential to address the resource allocation problem in the context of interdependent critical infrastructures for better mitigation planning.

The optimization of resource allocation in interconnected critical infrastructures is a topic that has been addressed extensively. For instance, Min, et al. [START_REF] Min | Toward modeling and simulation of critical infrastructure interdependencies[END_REF] have presented an integrated system to model the physical and financial impacts attributed to critical infrastructure interdependencies. Their framework comprises a system dynamics model, functional model and a non-linear optimization model. The purpose of the system dynamics model is to analyze the interdependencies between individual infrastructure components. The functional model is used to define the data requirements and the information exchanged between the models. The non-linear model enables the determination of optimal values of the control variables. The purpose of the work is to enable officials to respond to potential disruptions in a timely and effective manner. O'Reilly, et al. [START_REF] O'reilly | Interinfrastructure simulations across telecom, power and emergency services[END_REF] have specified a system dynamics model that describes the interactions between interconnected critical infrastructures. They use the model to analyze the impact of a telecommunications infrastructure failure on emergency services. The important conclusion is that lost communications negatively impacts medical services and drastically increases treatment costs.

Similarly, Arboleda, et al. [START_REF] Arboleda | Impact of interdependencies between infrastructure systems in the operation of health care facilities during disaster events[END_REF] have addressed the impact of failures of interdependent infrastructure components on the operation of healthcare facilities. The goal was to determine the unsatisfied demand of interconnected infrastructure systems and the resulting costs using a network flow model. Linear programming was used to assess the level of interdependency between a healthcare facility and the primary infrastructure systems linked to it.

In other work, Arboleda and colleagues [START_REF] Arboleda | Simulation as a tool to assess the vulnerability of the operation of a health care facility[END_REF] examined the internal operating capabilities of healthcare facilities in terms of the interactions between different service areas (emergency room, intensive care unit, operation room and wards). This was performed using a system dynamics simulation model. The goal was to assess the vulnerabilities of a healthcare facility during a disaster. The approach enabled the identification of policies to best mitigate the effects of a disruption.

Arboleda, et al. [START_REF] Arboleda | Vulnerability assessment of health care facilities during disaster events[END_REF] have also integrated a network flow model and system dynamics model. This was done to simulate the impact of infrastructure system disruptions on the provision of healthcare services.

These studies and others make it clear that wise decisions to reallocate and utilize the available resources are vital when dealing with interconnected critical infrastructures. Informed decisions can potentially mitigate death and devas-tation following natural or human-initiated catastrophes. The decisions must be made on the basis of sound knowledge and experience. In fact, the work described in this paper is motivated by the fact that decisions need to be carefully studied and pre-assessed before they are implemented. Moreover, they must be monitored and modified as the situation evolves. The next section discusses the application of agent-based models to address these issues.

Agent-Based Modeling

An agent-based model is a system of multiple autonomous decision making entities called agents. The agents are capable of sensing and interacting with each other within a modeled environment based on a set of predefined rules. The rules govern the behavior of the modeled agents and enable them to perform appropriate actions.

Agent-based modeling offers three key advantages in the context of realworld applications [START_REF] Bonabeau | Agent-based modeling: Methods and techniques for simulating human systems[END_REF]. First, it can capture emergent behavior that results from the interaction of individual entities (agents). Second, it facilitates detailed system descriptions by modeling and simulating the behavior of interacting entities. Third, it provides great flexibility to tune the complexity of individual entities to scenarios of interest.

These advantages have encouraged the application of agent-based modeling approaches by the disaster response community, the objective being to enhance disaster mitigation efforts. Atanasiu and Leon [START_REF] Atanasiu | Agent-based risk assessment and mitigation for urban public infrastructure[END_REF] have developed a multi-agent system based risk assessment tool for seismic hazards. Their tool, which incorporates an adaptive knowledge base, is designed to help create a risk management plan for better earthquake safety and response. The approach simulates emergency response actions for a set of earthquake scenarios at different urban locations. The results, which are displayed using a geographic information system (GIS), helps improve the quality of decision making. The decisions are typically made post-event for restoration and recovery operations aimed at rehabilitating the damaged infrastructure.

Thapa, et al. [START_REF] Thapa | Agent based decision support system using reinforcement learning under emergency circumstances[END_REF] have proposed an agent-based model for patient information acquisition and real-time decision making during emergencies. The model seeks to promote timely diagnosis and treatment of high-risk patients during emergency situations. The approach engages reinforcement learning in conjunction with an embedded dynamic programming mechanism to evaluate and improve a system value function and its policy.

Reinforcement Learning

Applications of reinforcement learning in agent-based models have attracted the interest of the critical infrastructure research community. The machine learning technique enables an agent to gain experiential knowledge by interacting with a massive number of disaster scenarios. The trained agent is then able to assist in disaster mitigation.

Wiering and Dorigo [START_REF] Wiering | Learning to control forest fires[END_REF] have developed an intelligent system that enables decision makers to mitigate the consequences of natural and human-initiated disasters (e.g., forest fires). Such disasters involve many interacting sub-processes that make it difficult for human experts to estimate costs. The system of Wiering and Dorigo uses reinforcement learning to learn the best policy or actions to be chosen in a variety of simulated disaster scenarios.

Su, et al. [START_REF] Su | Path selection in disaster response management based on Q-learning[END_REF] have proposed a path selection algorithm for disaster response management. The algorithm is designed for search and rescue activities in dangerous and dynamic environments. The algorithm engages reinforcement learning to help disaster responders discover the fastest and shortest paths to targeted locations. To accomplish this, a learning agent interacts with a twodimensional geographic grid model. After a number of trials, the agent learns how to avoid dangerous states and to navigate around inaccessible states.

Intelligent Decision Making

This section discusses how a reinforcement learning agent can be used for resource allocation in simulated interdependent critical infrastructures. The scenarios are modeled using i2Sim, a hybrid discrete-time simulator, which can handle vast numbers of interactions with the reinforcement learning agent. The simulated environment is based on an urban community similar to the Downtown Vancouver Model [START_REF] Khouj | Decision assistance agent in real-time simulation[END_REF]. The model incorporates four electrical power substations (P1, P2, P3 and P4), a water pumping station (W) and infrastructure assets such as venues (V1 and V2) and hospitals (H1 and H2). The continued interactions enable the agent to learn, improve its performance and make optimal decisions.

i2Sim

i2Sim is a hybrid discrete-time simulator that combines agent-based modeling with input-output production models. The simulator can model and play out scenarios involving interdependent systems. i2Sim is designed as a realtime simulator that can also serve as a decision support tool while a disaster is actually occurring. The simulation capability of i2Sim enables decision makers to evaluate the predicted consequences of suggested actions before they are executed [START_REF] Marti | Dynamic recovery of critical infrastructures: Real-time temporal coordination[END_REF].

The dynamic aspects of an i2Sim model are implemented by the movement of tokens between i2Sim production cells (i.e., modeled infrastructures such as power stations) through designated channels (i.e., lifelines such as water pipes). In fact, i2Sim cells and channels correspond to discrete entities in the real world. Figure 2 presents an example i2Sim model.

In i2Sim, each production cell performs a function. A function relates the outputs to a number of possible operating states -physical modes (PM) and resource modes (RM) -of the system. At every operating point along an event timeline, the i2Sim description corresponds to a system of discrete time equations expressed as a transportation matrix (Figure 3). The transportation matrix shows the interdependencies between the simulated quantities. In particular, the matrix in Figure 3 relates input quantities (XP1, XP2, ..., XS8, XS9) that arrive at the cells with the quantities that are produced as outputs of other cells (YP10, YP11, ..., YS17, YS18). These outputs can be distributed (via distributors) or aggregated (via aggregators) before being supplied to other cells. For instance, a water pumping station depends on water and electricity that are supplied by other cells (water supply and electrical power station). In row W5 of the transportation matrix, XW5 (water arriving at cell 5) comprises water that outputs from cells YW13, YW14 and YW15 (through X coefficients (internal links)) and power that outputs from cell YP10 (through Y coefficients (interdependent links)). The pumped water is distributed to a number of interconnected critical infrastructures [START_REF] Marti | i2Sim modeling and simulation framework for scenario development, training and real-time decision support of multiple interdependent critical infrastructures during large emergencies[END_REF].

Reinforcement Learning

Reinforcement learning [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] is a machine learning technique based on interactions between an agent and its environment. These interactions enable a reinforcement learning agent to maximize a time-delayed goal in the presence of uncertainty. Reinforcement learning occurs through the accumulation of experience, with the goal of finding actions that yield the greatest long-term rewards.

The actions taken in a given situation are determined by a policy realized by an action-value function. In general, reinforcement learning provides three ways of learning the policy: (i) dynamic programming; (ii) Monte Carlo estimation; and (iii) temporal difference. Monte Carlo estimation and temporal difference are the favored methods because they are model free. We have chosen to employ Monte Carlo estimation because it is well suited to learning from episodic problems of the type encountered in the disaster mitigation domain. Experimental results involving similar work [START_REF] Khouj | Decision assistance agent in real-time simulation[END_REF] reveal that convergence using step-by-step updates as prescribed by temporal difference learning take 2.6 times longer than episode-by-episode based updates as used in Monte Carlo estimation. The goal of the learning agent is to approximate the optimal action-value function leading to the best long-term reward that corresponds to the best trajectory. This recursive-learning algorithm uses incremental episode-by-episode back-ups to solve the well-known Bellman equation [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF].

The back-up formula is defined by the following equations for terminal and non-terminal states, respectively:

Q(s, a) ← Q(s, a) + α[R I + γR T -Q(s, a)] (terminal) (1)
Q(s, a) ← Q(s, a) + α[R I + γmax a Q(s ′ , a ′ ) -Q(s, a)] (non-terminal)(2)
where Q(s, a) is the action-value function of the current state-action pair; Q(s ′ , a ′ ) is the action-value function of the next state-action pair; α is the learning rate (i.e., extent to which the newly-required information overrides old information); R I is the immediate reward; R T is the terminal reward; and γ is the discount rate (i.e., influence that future rewards have on the learning process).

In Monte Carlo estimation, the back-up equation is used to recursively apply the terminal reward starting at the terminal state and back-stepping all the way to the start state. The estimate is computed by averaging the samples that are returned.

The action-value function can be implemented as a lookup table. The table associates a long-term predicted reward Q(s, a) value with each state-action pair defined for the modeled system. The table represents the acquired experience of the reinforcement learning agent and is updated during the learning process.

Note that the simulated system presents the state of the modeled environment that is detected by the learning agent. In the example considered here, the state is defined using two critical infrastructures: Power Station 4 and the Water Pumping Station. The physical mode (PM) and the resource mode (RM) of Power Station 4 and the Water Pumping Station are specified as PMXP4 and RMYP4 for power, and PMXW and RMYW for water. The values of X and Y range from one to five. When X has a value of one, the modeled infrastructure has no physical damage; when X is equal to five, the modeled infrastructure has collapsed completely. Similarly, when Y has a value of one, all the required resources to maintain the minimum functionality of the modeled infrastructure are available; when Y is equal to five, the required resources are not available.

Table 1 presents a sample lookup table used by the learning agent. In the table, the state-action pairs are captured using the variables: PMXP4, the Power Station 4 physical mode (state); RMYP4, the Power Station 4 resource mode (state); PMXW, the Water Pumping Station physical mode (state); RMYW, the Water Pumping Station resource mode (state); DP4, the Power Station 4 distributor (action); and DW, the Water Pumping Station distributor (action).

RL-MC Based Learning

The primary contribution of this paper is the application of reinforcement learning with Monte Carlo estimation (RL-MC) to problems involving interconnected and interdependent critical infrastructures. In the RL-MC approach, the problem is formulated as follows: the operating mode (physical mode and resource mode) of each modeled infrastructure unit represents the state of the targeted system. The distribution ratio of the available resources (associated with every modeled critical infrastructure) represents the actions that the agent can perform at every visited state. Every state-action pair is represented by a utility function that estimates the probability of obtaining the long-term reward upon choosing action a in state s. The estimate is computed by averaging the sampled returns over the long term. (In the scenario considered in this paper, the return is the expected number of discharged patients from two hospitals, H1 and H2). Terminal (R T ) and immediate rewards (R I ) are applied in the RL-MC approach.

The learning system (RL-MC) is implemented as a Java program that communicates with the simulator (i2Sim), which is realized in MATLAB (Figure 4). Communications are established via a software interface designed to support data transfer between MATLAB and the Java program. The interface allows the states of the simulated system and actions from the learning agent to be exchanged [START_REF] Khouj | Modeling Critical Infrastructure Interdependencies in Support of the Security Operations for the Vancouver 2010 Olympics[END_REF]. From i2Sim, the agent recognizes the state of the simulated environment based on the physical operability (PM) and the resource availability (RM) of the modeled infrastructures. The state is identified by the operating conditions of two critical infrastructures, Power Station 4 and Water Pumping Station (PMXP4, RMYP4, PMXW, RMYW). Accordingly, the agent selects the best distribution ratios (actions) for the distributors associated with Power Station 4 and the Water Pumping Station (DP4 and DW). The chosen action uses the distribution of the monitored resources (power and water) that maximizes the total number of discharged patients from hospitals H1 and H2.

The choice of Monte Carlo estimation over the temporal difference and dynamic programming approaches is also motivated by the need to reduce intrasystem communications in the architecture. In a reinforcement learning with temporal difference (RL-TD) approach, communications between the agent and i2Sim (Figure 4) introduce an overhead that is incurred at every time step [START_REF] Khouj | Decision assistance agent in real-time simulation[END_REF]. In fact, a significant portion of the computational time is due to the MATLAB-Java communications interface alone. In the case of RL-MC, the communications overhead occurs only twice per episode, once at the beginning and once at the end. Consequently, the communication time is reduced by almost a factor of three, which is advantageous when modeling complex systems.

RL-MC Algorithm

This section provides a technical description of how the RL-MC approach is realized within i2Sim.

In the RL-MC approach, the learning agent interacts with the modeled environment episodically. The agent follows a policy defined by the state-action value function. The agent attempts to learn an optimal policy. This sampling process terminates at a terminal state. At the terminal state, the estimation of the terminal state-action value function Q(s, a) is determined based on the total return that is observed at the end of each episode using Equation ( 1). This process averages the observed total returns of the visited states in the trajectory. For non-terminal states, the estimation of Q(s, a) occurs by backstepping Equation ( 2) to all state-action values in the sampled trajectory for each episode. In the limit, the learning agent successfully discovers the optimum trajectory.

The learning system implements the tabular form of the Q-learning algorithm using RL-MC. The lookup table is used to determine the action that is to be performed at the next state of the modeled environment. At any given state s, the learning agent performs an action a that delivers an adequate amount of resources (power and water) to the interconnected infrastructures. Note that 110 actions (N a = 110) and 225 states (N s = 225) are considered in modeled system. Each action a comprises instructions that specify the ratios of the five outputs of the P4 distributor (DP4) and the two outputs of the water distributor (DW). DP4 distributes power from Power Station 4 to five interconnected critical infrastructures: Hospital 1 (H1), Water Pumping Station (W), Venue 1 (V1), Power Station 2 (P2) and other interconnected infrastructures (Oth.). DW distributes the pumped water to the two modeled hospitals, H1 and H2.

The action vector a expresses the distribution ratios of the two distributors:

a = ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ DP 4 → H1 DP 4 → W DP 4 → V 1 DP 4 → P 2 DP 4 → Oth. DW → H1 DW → H2 ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ (3)
Given an initially-untrained lookup table, the RL-MC algorithm seeks to find the optimal action to perform in each state (optimum trajectory). If it is available, real-world experience could be used to initialize the lookup table as a starting estimate of the optimum schedule.

The environment state s is a vector that represents the operating conditions of the two modeled production cells (Power Station 4 and Water Pumping Station). The Power Station 4 and Water Pumping Station states are given by PMXP4 and RMYP4 for power and PMXW and RMYW for water, respectively. This can be represented at any given time by:

s = ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ P M1P
⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ (4)
The first row in the equation above corresponds to state The number of states N s in the model (total number of rows in vector s) is given by:

s 1 = (PM1P4, RM1P4, PM1W ,
N s = Z K = 15 2 = 225 states ( 5
)
where Z is the number of resource modes available for each controlled production cell; and K is the number of controlled production cells.

The number of available actions N a is given by:

N a = DP 4 × DW = 10 × 11 = 110 actions (6)
where DP 4 is the distributor associated with Power Substation 4; and DW is the distributor associated with the Water Pumping Station. The size of the lookup table L S is given by:

L S = N s × N a = 225 × 110 = 24, 750 rows. (7)
The states and actions, as described above, suggest a theoretical maximum lookup table size of 24,750 elements. As the simulation progresses, the history of actions and immediate rewards of each visited state (according to the policy) are accumulated. The immediate reward R I is applied at every time step by computing the difference in the discharged patients between the current and previous states:

R I = (N H1 + N H2 ) current -(N H1 + N H2 ) previous (8)
where N H1 is the number of discharged patients from Hospital 1; and N H2 is the number of discharged patients from Hospital 2. Note that the intermediate reward is a function of the number of patients discharged. The terminal reward R T is calculated and applied at the final time step only (terminal state) based on the total number of discharged patients from both hospitals:

R T = N H1 + N H2 . (9)
Each state-action value Q(s, a) is updated according to Equation (1) for a terminal state or according to Equation ( 2) for a non-terminal state.

At the end of the episode, according to the RL-MC algorithm, this information and the terminal reward R T are back-stepped through the sequence of state-action values performed during the episode.

Example Scenario

This section uses an example scenario to demonstrate the application of the learning agent to an urban community model simulated by i2Sim (Figure 5). The goal of the agent is to find the optimum trajectory that leads to the maximum outcome. The expectation is that this approach will converge quickly to the maximum number of discharged patients.

Environment Description

The simulated urban community model consists of nine interdependent critical infrastructure cells. The modeled cells are connected to each other through channels (e.g., underground cables, water pipes and roads). The resources generated by different cells are aggregated or distributed to other interconnected cells by control elements called aggregators and distributors such as power aggregators and water distributors. A pre-defined scenario defines the capacity and the operating parameters (input variables) of the modeled entities. The information is obtained from public domain data or directly from facility managers.

Four power cells are incorporated in the electricity infrastructure: Power Station 1, Power Station 2, Power Station 3 and Power Station 4. The cells determine the amount of power distributed to the system that comes in from the high-voltage supply system. The stations are geographically separated. Each power substation supplies a specific amount of power to its interconnected critical infrastructures. For example, Power Station 4 supplies 586 MW to its connected infrastructures (Hospital 1 and Water Pumping Station).

Similarly, the Water Pumping Station provides water to the connected hospitals (Hospital 1 and Hospital 2). The Water Pumping Station obtains power from Power Station 4 and water from an external source. The output of the cell is high pressure pumped water that goes to a water distributor, which distributes the water via water channels (water pipes).

Venue 1 and Venue 2 are facilities that contain large numbers of people. Venue 2 is more than 65,000 m 2 in area and hosts up to 60,000 people. Venue 1 is slightly smaller at about 44,000 m 2 and hosts up to 20,000 people. It is assumed that both venues are hosting events and are fully occupied. Thus, the total population is 80,000.

Two hospitals are modeled, Hospital 1 (main hospital) and Hospital 2 (alternative hospital). The input resources come from the four electrical power stations (electricity) and the water pumping station (water). Based on the availability of these resources, the rate of discharged patients for each hospital is known from historical data.

Scenario Description

The scenario was configured to reflect the damage caused by an earthquake. The simulated earthquake damaged Power Station 4. The physical structure of the power substation was not affected, but the resource availability RM was reduced due to a failure in one of the electrical feeders, RM2P4. Subsequently, as a result of the reduced electrical power, the water facility was not able to operate at full capacity and the power delivered to the venues and hospitals was also affected.

The earthquake produced casualties due to panic and chaos as people attempted to leave the venues. It was assumed that medical triage at the venues takes an average of 30 minutes per injured person. Upon completion of the assessment, emergency vehicles carried injured people to the emergency units at the hospitals for treatment. The travel time was assumed to be ten minutes.

At the emergency units, all the arriving patients were served on a first-comefirst-serve basis. Thirty minutes was assumed to be required to stabilize each trauma patient. The ability of the hospital to function at full capacity was, of course, impacted due to the limited supply of power and water.

The goal of the learning agent was to experience this scenario and to suggest a way to mitigate the impact on the hospitals. This was accomplished by adjusting the distribution ratios of the power and water distributors intelligently, as discussed in the next section.

Simulation Results

The simulations involved 100 scenarios per test, where each scenario represented a ten-hour period following the disaster event. Upon starting a scenario, the physical operability that represents the damage to the cells and channels was set to model a disaster. Following this, no further changes were made with regard to the extent of the damage. However, the available resources of the associated infrastructures change as the scenario evolves. The lookup table was The model simulated a period of ten hours in five-minute increments. The statistics and system latencies used by the simulator were taken from an internal technical report [START_REF] Khouj | Modeling Critical Infrastructure Interdependencies in Support of the Security Operations for the Vancouver 2010 Olympics[END_REF]. The report helped guide the rates used in the simulation. For example, a crowd of 80,000 is expected to have up to 480 injuries.

Figure 6 shows the results for two consecutive sets of trials that were initialized independently (light and dark lines). In both cases, the convergence to an optimum solution occurred and all 480 patients were discharged from both emergency units during the lifetime of the simulation.

During the early phases of learning in both trials, the agent had little or no experience and was unable to maximize the number of discharged patients. However, this was not the case in the later runs, where the agent showed an ability to fully satisfy the demands of the modeled interconnected critical infrastructures by carefully balancing resources across all the infrastructure components.

In contrast, a naive decision maker might select a resource configuration that would only favor the hospitals, but this would be a sub-optimal solution. Instead, the actions taken by the trained agent were those that intelligently utilized the available limited resources (power and water) without exhausting them, which ultimately satisfied the sudden needs of all the interconnected critical infrastructures, including the venues and, most importantly, the hospitals.

The simulation of each scenario required about three minutes using a computer with an Intel Core i5 2.8 GHz CPU and 8 GB RAM. In total, 100 runs took about 600 minutes. This is important in real-world deployments because simulations should be faster than real time in order to assist emergency responders in making informed decisions as a situation unfolds.

System Deployment

We envisage that the intelligent agent would be deployed as part of a larger software system aimed at providing decision assistance during actual emergencies. The software system would incorporate an i2Sim simulator, a learning agent, a library of pre-configured scenarios and an interface through which a user would interact with the system.

The suggested usage flow would first require the user to identify the disaster taking place in terms of the affected infrastructures. The system would provide a list of scenarios from which the user would pick the best match (instead of defining and entering a new scenario from scratch). In addition to the preconfigured scenario, the system would make available a pre-trained agent for the scenario. Should the scenario match be satisfactory, the human emergency responder can look to the agent for suggested actions in the situation at hand. If the scenario does not match the actual disaster, the user would have to manually adjust the scenario to accurately reflect the real-world situation. The pre-trained agent for the closest-matching scenario could still be used as a starting point.

Using a pre-trained agent is the best option for reducing agent learning time; learning by a trained agent is much faster than when an agent starts with a blank slate. A second approach relies on human knowledge acquisition. Important components of the knowledge and experience of trained emergency responders would have to be identified and captured. The output of this activity would be used to initialize the agent to reduce its learning time.

Conclusions

The modeling and analysis framework presented in this paper is an innovative approach for studying the impact of natural or human-initiated disasters on critical infrastructures and optimally allocating the available resources during disaster response. The framework relies on i2Sim and reinforcement learning using Monte Carlo policy estimation (RL-MC). i2Sim permits the simulation of complex interconnected critical infrastructures while the RL-MC approach supports rapid learning based on experiential knowledge in order to provide intelligent advice on allocating limited resources. The experimental results reveal that decision makers can reduce the impact of disruptions by employing the look-ahead and optimization features provided by the framework. The loosely coupled nature of reinforcement learning also enables it to be applied to a variety of resource optimization scenarios.

Our future research will analyze the computational aspects of the learning system. A speed versus accuracy trade-off exists between approaches that use the conventional lookup table implementation of an action-value function and other approaches that use function approximation techniques.

Figure 1 .

 1 Figure 1. Interconnected critical infrastructures.

Figure 2 .

 2 Figure 2. Example i2Sim model [11].

Figure 3 .

 3 Figure 3. Transportation matrix showing infrastructure interdependencies [11].

(

  < s >, < a >) Q ( s , a ) (< P MXP 4, RMY P 4, P MXW, RMY W >, < DP 4, DW >)

Figure 4 .

 4 Figure 4. MATLAB-Java dependency diagram.

  RM1W), where PM1P4 is the Physical Mode 1 of Power Station 4, RM1P4 is the Resource Mode 1 of Power Station 4, PM1W is the Physical Mode 1 of the Water Pumping Station and RM1W is the Resource Mode 1 of the Water Pumping Station.

Figure 6 .

 6 Figure 6. Agent learning behavior under RL-MC.

Table 1 .

 1 Sample lookup table (s: state, a: action).

Acknowledgement

This research was partially supported by the Ministry of Higher Education of the Kingdom of Saudi Arabia.