Carole Adam
email: carole.adam@imag.fr

Benoit Gaudou
email: benoit.gaudou@ut-capitole.fr

BDI agents in social simulations: a survey

Keywords: agent-based modeling and simulation, BDI agents, social sciences

Modeling and simulation have long been dominated by equation-based approaches, until the recent advent of agentbased approaches. To curb the resulting complexity of models, Axelrod promoted the KISS principle: "Keep It Simple, Stupid". But the community is divided and a new principle appeared: KIDS, "Keep It Descriptive Simple". Richer models were thus developed for a variety of phenomena, while agent cognition still tends to be modelled with simple reactive particle-like agents. This is not always appropriate, in particular in the social sciences trying to account for the complexity of human behaviour. One solution is to model humans as BDI agents, an expressive paradigm using concepts from folk psychology, making it easier for modellers and users to understand the simulation. This paper provides a methodological guide to the use of BDI agents in social simulations, and an overview of existing methodologies and tools for using them.

Introduction

After years dominated by equations, that describe the behaviour of the studied system or phenomenon only from a global point of view, modelling and simulation have undergone a deep revolution with the application of Multi-Agent Systems to the problem of the modelling and simulation of complex systems. Agent-Based Modelling and Simulation (ABMS) is a recent paradigm that allows modellers to reason and represent the phenomenon at the individual level, and to take into account heterogeneity and complexity both in the individual layer and in the environment layer. ABMS has been successfully used for example in Ecology [START_REF] Deangelis | Individual-Based Models and Approaches in Ecology[END_REF], Economics [START_REF] Tesfatsion | Agent-based computational economics: growing economies from the bottom up[END_REF] or Social Sciences [START_REF] Gilbert | Simulation For The Social Scientist -Second Edition[END_REF].

In the context of agent-based models aimed to "enrich our understanding of fundamental processes", Axelrod advocates the KISS principle: "Keep It Simple, Stupid" [START_REF] Axelrod | The Complexity of Cooperation: Agent-Based Models of Competition and Collaboration[END_REF]. His reasoning is that "both the researcher and the audience have limited cognitive ability", which limits their ability to understand the model content, and thus their level of confidence in the emergence of a "surprising result" [11, p. 5]. That is, by keeping the assumptions underlying the model simple, we can better understand the complexity which arises from the simulation. This notion of simplicity manifests itself in a model as: a small number of parameters, a simple environment, a small number of entities, and very simple (reactive) agent behaviour. But whether simplicity is always appropriate (and if not, when not) has come into question in the multi-agent community, see e.g. [START_REF] Drogoul | Multi-agent based simulation: Where are the agents?[END_REF]. Indeed it has been suggested that there is much to be gained from the simulation of richer, more realistic models; one perspective on this is captured by the KIDS principle (Keep It Descriptive, Stupid). This principle introduced by Edmonds et al. [START_REF] Edmonds | From kiss to kids: an anti-simplistic modelling approach[END_REF] recommends using a formalism as descriptive and straightforward as needed (and possible) when modelling, even if it implies greater complexity. The idea is that it is better to start from a well-adapted formalism, descriptive enough for the needs of the simulation, and to possibly simplify it later, rather than to start from a simple but unadapted formalism that then needs to be extended in an ad hoc way to meet the simulation needs. As an example of the KIDS approach, a lot of work has been done on environmental management (with Geographic Information Systems, e.g. [START_REF] Thierry | Managing agricultural landscapes for favouring ecosystem services provided by biodiversity: a spatially explicit model of crop rotations in the GAMA simulation platform[END_REF]) with a large number of parameters, heterogeneity among entities, precise time management, and multi-scale simulations.

Nevertheless, the behavioural component usually remains very simple, in particular in ecological or epidemiological simulations [START_REF] Laperriere | Structural validation of an individual-based model for plague epidemics simulation[END_REF][START_REF] Simoes | An agent-based/network approach to spatial epidemics[END_REF] that are interested in very simple entities (viruses, bacterias, ducks, etc.), which is their main reason for modelling them with simple and reactive agents. Sun says: "a significant shortcoming of current computational social simulation is that most of the work assumes very rudimentary cognition on the part of agents. [...] It not only limits the realism, and hence the applicability, of social simulations, but also precludes the possibility of tackling the question of the micro-macro link in terms of cognitive-social interaction" [START_REF] Sun | Cognitive social simulation incorporating cognitive architectures[END_REF].

Simulations would therefore benefit from more complex agents models. However, Edmonds and Moss admit that the suitability of the KIDS principle over KISS, i.e. the choice between descriptivity and simplicity, highly depends on the context, the goal of the simulation, and various parameters [START_REF] Edmonds | From kiss to kids: an anti-simplistic modelling approach[END_REF]. So finally, the KISS approach favouring simplicity over descriptivity is perfectly adapted in some types of simulations, while in other cases the simulation could greatly benefit from using a more descriptive agent model.

In this paper, we will focus on applications of ABMS to the field of Social Sciences, which is interested in more complex entities (namely human beings) in their complex social behaviour and in their interactions with each other and their environment. In that particular field of application, we want to discuss the suitability of more descriptive agent models to represent humans. Indeed, reactive particle-like agents in an electromagnetic field may be adapted in some contexts (e.g. to model crowds), but they are too often the first reflex of modellers even when they are not adapted, while other alternatives do exist and have many benefits. The advantages of more complex and/or descriptive models include keeping the programming intuitive, allowing agents to adapt to the environment, and having agents which can explain their behaviour.

There are several kinds of complex agents that can be used; in this paper we focus in particular on BDI agents because they offer a straightforward formalisation of the reasoning of human agents with intuitive concepts (beliefs, desires and intentions) that closely match how humans usually explain their reasoning. For example [102, p.1] argues that "BDI agents have been used with considerable success to model humans and create human-like characters in simulated environments. A key reason for this success is that the BDI paradigm is based in folk psychology, which means that the core concepts of the agent framework map easily to the language people use to describe their reasoning and actions in everyday conversations". Such a straightforward matching between the concepts being modelled and the formal concepts used in the model is exactly what is advocated by the KIDS principle. It also provides numerous advantages: "the close match between the core BDI concepts and the natural terminology that the experts being modelled used to describe their reasoning facilitated knowledge capture, representation and debugging" [102, p.1] Despite these arguments in favor of using BDI agents in social simulations, some features also deter from using them, in particular their complexity.

To summarise, in this paper we argue in favour of the KIDS approach in a particular context which makes it intuitively more adapted: social science simulations which are interested in human behaviour. In this field, the KIDS approach recommends the use of more complex and descriptive agent models. We focus in particular on BDI agents, since they use folk psychology concepts that straightforwardly match human reasoning as people understand it, making the models easier to design and to understand. However BDI agents are complex, heavy, and not always justified. It is therefore necessary to precisely identify the types of simulations where modellers should take advantage of the benefits they provide, and the other types of simulations where the inconveniences outweigh these benefits. We start with further motivating our focus on BDI agents and defining their features and benefits for social sciences simulations (Section 2). We then go on to provide a methodological guide for modellers regarding the use of BDI agents in their simulation (Section 3). We discuss the literature and compare existing simulations on different axis: the required characteristics in the agents (e.g. learning, emotions, Section 3.1); the field of application (Section 3.2); the goal of the simulation (e.g. training, knowledge discovery, Section 3.3); or the desired level of observation (Section 3.4). We also discuss various features that could deter from using BDI agents in simulations (Section 3.5). Finally Section 4 presents a short overview of available methodologies and tools for concretely integrating BDI agents into simulations.

Motivations for BDI agents in social simulations

Agent-based models present a great variety of agent architectures, from the simplest particle architectures to the most complex cognitive architectures. In this section we illustrate this variety with a few examples. We then justify our choice of the BDI architecture over other types of "intelligent" agent architectures, and finally give more details about BDI agents and their benefits.

Examples of ABMS with various kinds of agents

2.1.1 Simulation of escaping panic [START_REF] Helbing | Simulating dynamical features of escape panic[END_REF] Helbing et al. [START_REF] Helbing | Simulating dynamical features of escape panic[END_REF] propose a model of crowd behaviour in the case of emergency escape. They aim to model panic situations where crowd pressure can be dangerous and to find solutions for the evacuation of dangerous places (e.g. smokefilled rooms). The authors chose to represent the agents as particles in a force field: all physical and social interactions with other agents and the environment are represented as forces. The movement of particles representing the human beings are described by electromagnetic-like equations.

Simulation of pedestrian mobility after an earthquake

Truong et al. [START_REF] Van Truong | Developing a model of evacuation after an earthquake in lebanon[END_REF] propose a model of urban pedestrian mobility after an earthquake in Lebanon, in order to show the importance of population preparedness to reduce casualties. In a data-based geographical space composed of streets, buildings and green spaces, agents representing human beings behave in order to evacuate to the green spaces. Their behaviour is driven by a set of rules influenced by agent features (gender, age...) and by stimuli from the environment and from the other agents. [START_REF] Cecconi | Individual versus social survival strategies[END_REF] Cecconi and Parisi [START_REF] Cecconi | Individual versus social survival strategies[END_REF] propose the use of simulation to evaluate various survival strategies of individuals in a social group. Surviving (and reproducing) or dying depends on the resources available to the agent. These resources can be physical resources (such as food, money), other agents (such as a reproduction partner) or personal skills (such as fighting or working capability). Agents can use two types of strategies: Individual Survival Strategies and Social Survival Strategies; a particularly interesting social strategy consists in giving part of what the agent collects to supply a common pool of resources. The authors were interested not only in the evolution of skills over generations but also in the ideal size of the social groups. Each agent is represented as a neural network. The evolution is modelled by a genetic algorithm aimed at improving the capability to find and produce food.

Simulation of survival in a group

Crowd simulation for emergency response [133]

Shendarkar et al. [START_REF] Shendarkar | Crowd simulation for emergency response using BDI agents based on immersive virtual reality[END_REF] model a crowd evacuation under terrorist bomb attacks in a public area. They investigate the influence of various important parameters (e.g. density or size of the crowd, number of policemen driving the evacuation, number of exits in the room) on the quality of the evacuation (e.g. evacuation time, number of casualties). Agents are implemented with an extended BDI architecture, which includes an emotional component and a real-time planner. The behaviour of the agents is captured from a human evolving in a Virtual Reality laboratory (participative modelling).

2.1.5

Revamping the simulation of the survival in a group [START_REF] Sun | Social institution, cognition, and survival: a cognitive-social simulation[END_REF] Sun et al. [START_REF] Sun | Social institution, cognition, and survival: a cognitive-social simulation[END_REF] propose to give genuine cognitive capabilities to the agents involved in Cecconi's simulation [START_REF] Cecconi | Individual versus social survival strategies[END_REF], in order to make it more cognitively realistic. Moreover, they introduce social institutions that impose obligations on the agents and sanctions in case of violation; for example, an agent can be punished if it refuses to contribute food to the central store when required. Through these additions to the original model, the authors aim at studying the interactions between individual cognition and social institutions. The cognitive processes of the agents are implemented by using the CLARION cognitive architecture, mainly based on Sun's work in cognitive sciences [START_REF] Sun | From implicit skills to explicit knowledge: A bottom-up model of skill learning[END_REF]. It implements mechanisms for implicit vs explicit learning, among numerous other cognitive capabilities.

Summary

These few examples show a huge variety in the agent architectures. Indeed, they illustrate respectively: a particle architecture; a rule-based architecture; a neural network architecture; a BDI architecture; and a cognitive architecture. We can make a dichotomy between these five architectures w.r.t. the concepts manipulated to implement the behaviours and their complexity level.

The first three architectures (particle, rule-based and neural network) do not involve any cognitive concepts in the reasoning and decision-making process. Even if agents represent human beings, their behaviours are for the most part a function of the external stimuli (e.g. the external forces in the first example, the perception of the environment in the second and third examples). These architectures are traditionally called reactive agents [START_REF] Wooldridge | An Introduction to MultiAgent Systems -Second Edition[END_REF]. They are very well adapted to represent simple entities without complex reasoning capabilities (e.g. ants) or to model human beings when focusing on simple behaviour, but they cannot deal with complex reasoning mechanisms involved in more complex human behaviour.

In contrast, the last two architectures represent two kinds of intelligent agents [START_REF] Wooldridge | An Introduction to MultiAgent Systems -Second Edition[END_REF]: BDI agents and agents with cognitive architectures1 . Intelligent agent architectures allow agents to be more autonomous and proactive, not only reacting to environmental stimuli, therefore matching humans more closely. They enable the modelling of more complex issues than would be possible with purely reactive agents (as shown by the progression between 2.1.3 and 2.1.5), making them very relevant to our aim of integrating much more descriptive agents in social simulations. Finally their strong theoretical grounding can provide many guidelines in the modelling of agents, but also many constraints.

Choice of the agent architecture

Another survey of agent architectures

Balke and Gilbert [START_REF] Balke | How do agents make decisions? a survey[END_REF] recently published a survey of fourteen decision-making architectures for agents, from very simple to very complex, aiming at providing guidelines regarding which one of them should be used for a given research question. One of the architectures they study is the BDI one, some others are various extensions of the standard BDI (with emotions, norms...).

On the contrary in this paper, we focus only on the BDI architecture. We started from the observation that the agents used in simulations are too simple, and more descriptive agents would bring many benefits in realism and validity of results. In the next paragraph we explain why we focus on BDI rather than cognitive architectures such as SOAR or ACT-R.

Our goal is therefore different in that we specifically argue for the wider use of BDI agents in social simulations. We also go beyond Balke and Gilbert's review by providing a technological guide showing interested designers not only why they should use BDI agents, but also how to use them. Indeed, even when designers are aware of the benefits that BDI agents could bring in their simulation, they are often limited by technical difficulties, in particular a lack of methodologies and tools.

BDI vs cognitive architectures

Both BDI architectures and cognitive architectures allow modelers to deal with the cognitive processes of the agents and to provide the descriptivity we (after [START_REF] Edmonds | From kiss to kids: an anti-simplistic modelling approach[END_REF]) claim is needed in some social simulations. However these two kinds of architectures are very different.

First of all, they differ on their conceptual inspiration, i.e. the origin of the underlying concepts. Cognitive architectures are based on cognitive sciences and aim at describing human cognitive processes as precisely as possible. This requires in-depth analysis of the functioning of the human brain from a biological or neurological point of view (for example to determine the area of the brain that performs a given function). In contrast, the BDI architecture is based on a philosophical stream. Bratman's work [START_REF] Bratman | Intentions, Plans, and Practical Reason[END_REF] is formalised mainly with modal logics [START_REF] Rao | Modeling rational agents within a BDI-architecture[END_REF][START_REF] Cohen | Intention is choice with commitment[END_REF]. These logics describe the three main intentional states (Beliefs, Desires, Intentions) that lead the agent from its world knowledge, to the choice of the action to perform, and to the execution of these actions. The BDI architecture therefore offers a more straightforward description making models easier to understand for modellers and end-users.

As a result, BDI and cognitive architectures also differ on the abstraction level of the concepts manipulated. BDI concepts are higher-level concepts (mental attitudes), that actually result from the low-level concepts and principles described in cognitive architectures (e.g. memory chunks in ACT-R). According to Sun, BDI architectures can be viewed as the explicit and symbolic part of cognitive architectures. BDI architectures focus on the conscious level of one's mind, at a level where human beings can consciously manipulate the concepts and communicate them to others [START_REF] Norling | Folk psychology for human modeling: extending the BDI paradigm[END_REF]. This allows the easy mapping of knowledge from experts to agent behaviour, and helps non-computer scientists to understand agent behaviour.

Finally, they differ on their degree of theoretical standardisation. Each cognitive architecture implementation builds upon its own theory: for example CLARION [START_REF] Sun | From implicit skills to explicit knowledge: A bottom-up model of skill learning[END_REF] is based on Sun's work [START_REF] Sun | Duality of the Mind[END_REF]; SOAR [START_REF] Rosenbloom | The SOAR papers: research on integrated intelligence[END_REF] is based on Newell's work [START_REF] Newell | Unified theories of cognition[END_REF]; and ACT-R [START_REF] Anderson | The atomic components of thought[END_REF] is based on Anderson's work [START_REF] Anderson | The architecture of cognition[END_REF][START_REF] Anderson | Rules of the mind[END_REF]. They have no common focus: CLARION is interested in the dichotomy implicit vs explicit, in particular for the learning process, whereas ACT-R focuses on the activation process. There is no consensus either on the memory component of the architecture, be it on the number of memories or their nature. On the contrary, in all of the substantial number of existing implementations of the BDI framework and formalisations of the underlying modal logic, the BDI trinity is a constant, allowing a huge community of researchers to work on its improvement and extensions.

Sun has argued that cognitive architectures can enhance simulations [START_REF] Sun | Cognition and multi-agent interaction: from cognitive modeling to social simulation[END_REF], and proved his point by revamping an existing simulation with a cognitive architecture instead of the initial reactive agents [START_REF] Sun | Cognitive social simulation incorporating cognitive architectures[END_REF]. We argue that BDI architectures can similarly enhance simulations, and also provide additional benefits as shown by Norling: "the close match between the core BDI concepts and the natural terminology that the experts being modelled used to describe their reasoning facilitated knowledge capture, representation and debugging" [102, p.1].

Therefore BDI architectures appear to be both descriptive enough to describe cognitive processes influencing behaviour, and intuitive enough to be easily understood by modelers and non-computer-scientists in general. For these reasons, in this paper we are more particularly interested in BDI agents, and want to provide a guide as to when they can and should be used in social sciences simulations. A similar guide could be proposed about when to use cognitive architectures, but this is not the scope of this article.

Description of a BDI agent

BDI Theory

The BDI model is theoretically grounded in the philosophical work of [START_REF] Bratman | Intentions, Plans, and Practical Reason[END_REF] and [START_REF] Dennett | The intentional stance[END_REF] where the three basic components (beliefs, desires and intentions) are defined. The model attempts to capture the common understanding of how humans reason through: beliefs which represent the individual's knowledge about the environment and about their own internal state; desires or more specifically goals (non conflicting desires which the individual has decided they want to achieve); and intentions which are the set of plans or sequence of actions which the individual intends to follow in order to achieve their goals.

Cohen and Levesque distinguish BDI from purely reactive models by adding a level of commitment to the set of intentions to achieve a long-term goal [START_REF] Cohen | Intention is choice with commitment[END_REF]. This notion of commitment is therefore a critical part of the BDI model. Rao and Georgeff show there must also be some form of rational process for deciding which intentions to select given a set of circumstances [START_REF] Rao | Modeling rational agents within a BDI-architecture[END_REF]. While the specific way in which this is done varies among implementations, it is generally guided by folk psychology principles [START_REF] Norling | Folk psychology for human modeling: extending the BDI paradigm[END_REF].

So the core functionalities that BDI systems should implement are: a representation of beliefs, desires and intentions; a rational and logical process for selecting intentions; and a flexible and adaptable commitment to the current set of intentions.

Implemented systems

There are a substantial number of implemented BDI systems (e.g. JACK [START_REF] Busetta | Jack intelligent agents-components for intelligent agents in java[END_REF], Jason [START_REF] Bordini | Programming multi-agent systems in AgentSpeak using Jason[END_REF], 3APL [START_REF] Hindriks | Agent programming in 3APL[END_REF], Gorite [START_REF] Rönnquist | The goal oriented teams (gorite) framework[END_REF], JADEX [START_REF] Pokahr | Jadex: A bdi reasoning engine[END_REF], etc) supporting the creation of agents defined using these three constructs, and several systems extending these to provide additional human reasoning capabilities. Common to all these implementations is the concept of a plan library which is a collection of plans that the agent is capable of performing. Each of these plans consists of a body, a goal which the plan is capable of achieving, and a context in which the plan is applicable. The plan body can consist of both: actions which can affect the environment directly, and subgoals which will be expanded into other plans when needed. This nesting of goals within plans allows for a plan-goal hierarchy structure to be constructed, with branches from a goal node leading to plans capable of achieving this goal, and branches from a plan node leading to goals which must be achieved to complete this plan.

Implemented BDI systems also contain a BDI execution engine, which guides the agent reasoning process. There are some variations in the algorithms used by different engines, but a basic version is outlined below:

1. Perception: find any new events which may have been triggered, either within the agent or externally from the environment;

2. Update: update beliefs with new information provided by this event;

3. Intention revision: if the change in beliefs means that an intention or goal is no longer valid (either because it has already been achieved or is no longer possible) then remove it from the set, which may result in some form of failure recovery by attempting to achieve it with a different plan;

4. Plan filtering: if the intention was revised, determine the set of applicable plans for the current event, which are appropriate given the current context, and are defined as capable of achieving the goal associated with the event;

5. Plan selection: if there is no current plan, select a new plan from this set (the selection method varies from selecting the first available plan to selecting the plan with the greatest expected utility) and add the components of its body to the intention list;

6. Action: execute the next step of the current plan, which may involve executing an action or expanding a subgoal by triggering a new event.

Core features and benefits

The BDI architecture therefore offers the following desirable core features:

• Adaptability -Plans are not fully specified and can consist of nested subgoals, which allows flexibility in the agent, making it able to adapt to a changing environment

• Robustness -The plan hierarchy structure means that if a plan fails, potentially because of environment changes during its execution, the agent is able to recover by executing another applicable plan if one is available.

• Abstract programming -The programmer specifies beliefs, desires and intentions at a high level, making it possible to create complex systems while maintaining transparent, easily understandable code.

• Explainability -Using a goal-based approach as opposed to a task-based approach means that an agent knows why it is doing a specific task. This allows agents to explain their behaviour to the modeller or the user in an intuitive manner.

BDI extensions

Beyond these core features, the BDI framework can be extended to provide additional desired features, and many works were dedicated to such extensions. For instance Singh et al. extend BDI with learning [START_REF] Singh | Integrating learning into a bdi agent for environments with changing dynamics[END_REF], in order to improve plan selection by dynamically adjusting a confidence measure. There has also been work for integrating real-time planners into the BDI architecture to support the creation of plans at runtime using First Principles [START_REF] Silva | First principles planning in bdi systems[END_REF] among other techniques. Noir et al. [START_REF] Nair | The role of emotions in multiagent teamwork[END_REF] use a BDI model of coordination, combined with POMDP to handle uncertainty, and prove that their agents perform better teamwork.

A large panel of social concepts have been added to the basic BDI framework. Dignum et al. [START_REF] Dignum | Towards socially sophisticated BDI agents[END_REF] introduce the concept of norms to "assist in standardising the behaviour of individuals, making it easier to cooperate and/or interact within that society". In the same way human behaviour is very much influenced by social norms, agent behaviour can be influenced by norms too, allowing agents to have a better understanding of what other agents are likely to do. Gaudou [START_REF] Gaudou | Formalizing social attitudes in modal logic[END_REF] focuses on the introduction of social attitudes (and in particular group belief), and Lorini et al. [START_REF] Lorini | The logic of acceptance: grounding institutions on agents attitudes[END_REF] go one step further by modelling the group acceptance in an institutional context. Trust has also been introduced in several BDI works: Taibi [START_REF] Taibi | Incorporating trust into the bdi architecture[END_REF] has implemented trust in a BDI interpreter to help agents to delegate tasks; while Koster et al. [START_REF] Koster | Opening the black box of trust: reasoning about trust models in a bdi agent[END_REF] allow the agent to reason about its trust model and to adapt it to a given context.

Adam et al. extend the BDI framework with emotions by formalising their triggering conditions [START_REF] Adam | OCC's emotions: a formalization in a BDI logic[END_REF][START_REF] Adam | A logical formalization of the OCC theory of emotions[END_REF], and propose an associated reasoning engine for BDI agents [START_REF] Adam | A bdi emotional reasoning engine for an artificial companion[END_REF]. Other works had the same aim: Pereira et al. present an extension to the BDI architecture to include Artificial Emotions [START_REF] Pereira | Towards an architecture for emotional BDI agents[END_REF] affecting an agent's reasoning in a similar way to human emotions; and Jones et al. also propose an extended BDI architecture integrating emotions [START_REF] Jones | Personality, emotions and physiology in a bdi agent architecture: The pep -bdi model[END_REF].

3 Methodological guide: when to use BDI agents in simulations?

In this section, we aim at providing a methodological guide to help modelers to choose whether to use a BDI architecture or not, depending on the desired agents characteristics (Section 3.1), on their field of application (Section 3.2), on the goal of their simulation (Section 3.3), or on the target observation level (Section 3.4). We also discuss some arguments against the use of BDI agents, along with possible answers (Section 3.5).

Influence of the characteristics of the agents

Representation and reasoning

The BDI framework is a model of how humans represent their perceived environment in terms of mental attitudes, and how they reason about it to deduce new mental attitudes from their existing ones. Works in modal logic have investigated these concepts in details, and provided formalisations of their links with each other and of various deduction rules. Modal logic can be seen as a strict and constraining formalism, but it has the advantage of being totally unambiguous.

The BDI model allows a subjective representation of the environment in terms of beliefs, that can be incomplete, erroneous (due to flawed perceptions), or differ between agents. For example in the AMEL model of evacuation after an earthquake, each agent has their own map of the roads and position of obstacles [START_REF] Van Truong | Developing a model of evacuation after an earthquake in lebanon[END_REF]. To account for the uncertainty of perceptions in unknown or partially known environments, Farias et al. have extended BDI agents with fuzzy perception [START_REF] Farias | Bdi agents with fuzzy perception for simulating decision making in environments with imperfect information[END_REF].

Another major benefit is the possibility for the agent to represent and to reason about other agents' mental attitudes (Theory of Mind, [START_REF] Goldman | Theory of mind[END_REF]). For example, Bazzan et al. [START_REF] Bazzan | Agents in traffic modelling: from reactive to social behaviour[END_REF] have developed a model of traffic in which (social BDI) drivers can reason about other drivers' behaviour when making their decision. Similarly Bosse et al. [START_REF] Bosse | A two-level bdi-agent model for theory of mind and its use in social manipulation[END_REF] have developed a BDI model allowing agents to reason about other agents and their reasoning process, and use it to represent a manager's manipulation behaviour over his employees.

Such a capacity to represent and reason about the world in terms of mental attitudes can be used to develop agents that can explain their behaviour in understandable terms to human modellers and users: these are known as self-explaining agents [START_REF] Harbers | Self-explaining agents[END_REF][START_REF] Fähndrich | Self-explaining agents[END_REF].

Self-explaination of behaviour

The issue of explaining a reasoning is as old as expert systems [START_REF] Swartout | Explanations in knowledge systems: design for explainable expert systems[END_REF], a field where a lot of work has been done to develop explanatory systems and methodologies. In particular, when an expert system is used in a learning environment (for instance a military training environment [START_REF] Gomboc | Design recommendations to support automated explanation and tutoring[END_REF]), it is very important for the users to be able to get explanations about the reasoning made by the system, and about how and why it behaved as it did. More generally, Ye and Johnson have shown that an expert system explaining its response was better accepted by end-users [START_REF] Ye | The impact of explanation facilities on user acceptance of expert systems advice[END_REF]: indeed if the system can justify its choices, then it will increase the end-user's trust. This issue is still current in Multi-Agent Systems [START_REF] Fähndrich | Self-explaining agents[END_REF].

In agent-based simulations, the problem is quite similar: a simulator used to teach something is more effective if it is able to give explanations of its behaviour. More generally, Helman et al. [START_REF] Helman | Explanation systems for computer simulations[END_REF] have shown that explanations of computer science simulations are useful for modellers to better understand the simulation. For example we can assume that an agentbased simulation used as a decision support tool by policy-makers (e.g. in urban planning, ecological risk assessment, etc) would be better understood and accepted, and as a consequence more efficient, if some agents (and in particular those representing human beings or institutions) could be asked for explanations.

The introduction of BDI agents in the simulation is particularly interesting from this point of view. For example Harbers et al. [START_REF] Harbers | Explaining simulations through self-explaining agents[END_REF] have investigated the usefulness of BDI agents to provide explanations to the learner in a training simulation. Indeed an interesting capability of BDI agents is that they can explain their behaviour in terms of their goals (why they did what they did) and their beliefs (what were the reasons for their choice of action); and these explanations can become much more precise by adding complementary concepts (e.g. norms) to the basic BDI model. Moreover the concepts involved in these self-explanations are very intuitive for human beings because they match their own natural language and their way of thinking [START_REF] Norling | Folk psychology for human modeling: extending the BDI paradigm[END_REF], making explanations provided by BDI agents more effective.

Therefore, BDI concepts facilitate the creation of explanations. Nevertheless, the counterpart of this gain in terms of explanation power is a much more complex reasoning process. This is not necessarily a drawback against the use of BDI agents, on condition that these agents are systematically endowed with a self-explaining capability. Indeed, as highlighted in [START_REF] Harbers | Explaining simulations through self-explaining agents[END_REF], without providing this capability to agents, the agents behaviour and therefore the whole simulation become almost impossible to understand, both by computer scientists implementing it and by modellers trying to use it. To put it differently, a side-effect of the introduction of BDI agents in simulations is the necessity to systematically endow agents with a clear self-explanation capability in order to make the simulation understandable, but the implementation of this self-explanation capability is facilitated by the BDI concepts underlying the reasoning of these agents.

Complex interactions via communication

In the agent-oriented programming community, in parallel to the development of the BDI architecture many efforts have been dedicated to the development and formalisation of Agent Communication Languages (ACL) (e.g. KQML [START_REF] Finin | Kqml as an agent communication language[END_REF] or FIPA-ACL [START_REF]FIPA Communicative Act Library Specification[END_REF] to cite the most important ones) and Interaction Protocols (IP) (e.g. the FIPA Contract-Net Interaction Protocol [START_REF] Fipa | FIPA Contract Net Interaction Protocol Specification[END_REF]).

Such ACLs and IPs have two main advantages, that can be directly applied to simulation. First of all they allow agents to communicate in a realistic (human-like) way, with a large choice of speech primitives. The communication is closely linked to the mental states of the agents by the ACL semantics (for example the FIPA-ACL one [START_REF]FIPA Communicative Act Library Specification[END_REF]). Therefore this allows one to easily model the effect of the communication on the agent behaviour: received messages modify the mental states of the agent, which in turn have an influence on its behaviour. For example, Fernandes et al. propose a traffic simulator in the context of Intelligent Transportation Systems [START_REF] Fernandes | Multi-agent architecture for simulation of traffic with communications[END_REF], with communication between vehicles. Thus the simulator must take into account this communication and its influence on the vehicle driver's behaviour. The authors follow the Prometheus modelling methodology [START_REF] Padgham | Prometheus: A methodology for developing intelligent agents[END_REF] to integrate BDI agents and communication via FIPA-ACL.

The second interest of using complex communication in the simulation is the use of the Interaction Protocols. Some protocols have been formalised, in particular by the FIPA consortium, in order for agents to coordinate their tasks. This is particularly useful when the simulation manages a swarm of agents such as in rescue simulations. Some simulation platforms integrate these mechanisms for this purpose. For example, Sakellariou et al. [START_REF] Sakellariou | Enhancing netlogo to simulate bdi communicating agents[END_REF] have developed a forest fire simulator where firemen agents cooperate and coordinate under the Contract Net Protocol. To this purpose, they have extended the NetLogo platform [163] with a tool for easily integrating BDI agents exchanging FIPA-ACL messages in a FIPA protocol. In addition, Thabet et al. [START_REF] Thabet | Towards an adaptive grid scheduling: Architecture and protocols specification[END_REF] show on a grid scheduling example that a MAS involving BDI agents communicating with high-level FIPA-ACL can bring benefits on two levels: at the micro level, agents are able to perceive the network, discover failures and reason about them to choose the most adapted behaviour; at the macro level, agents use high-level Interactions Protocols, providing them with a powerful and flexible way to communicate.

Complex communication mechanism with BDI agents are also used to model negotiation behaviours. Such models have been used for instance to simulate negotiation in mix human-agents teams in virtual worlds for training [START_REF] Traum | Negotiation over tasks in hybrid human-agent teams for simulation-based training[END_REF]. The aim of this simulation is to improve the leadership skills of human beings by interacting with virtual teammates as believably as possible.

To conclude, when using BDI agents, the use of complex communication with ACL and IP is quite easy, in particular because both share the same philosophical ground and the same concepts. This provides, when it is needed in the model, powerful coordination, cooperation or negotiation mechanisms via communication.

Planning

Plans are sequences of actions that an agent can perform to achieve its goal(s). In some simulation scenarios a modeller may want agents to continuously rethink their behaviour, based on their situation and goal(s), in a manner that cannot be captured by simple if-then rules. This may involve, for example, modelling a complex decision process, supplying the agent with a comprehensive library of plans, or equipping an agent with fundamental planning capabilities.

Examining the application of multi-agent simulation even in just one domain (crisis response) we already find agents equipped with a broad range of planning capabilities. PLAN C2 uses reactive agents with simple scripted plans to model medical and responder units in a mass casualty event simulations to explore the medical consequences of possible response strategies. DrillSim [START_REF] V. Balasubramanian | Drillsim: A simulation framework for emergency response drills[END_REF] model fire warden and evacuee agents with a recurrent artificial neural network to select their course of action and an A* algorithm for low-level path planning; these agents are used to test the benefits of information collection and dissemination in a fire evacuation scenario. D-AESOP [START_REF] Buford | D-aesop: a situation aware bdi agent system for disaster situation[END_REF] models medical relief operations in crisis situations, in an attempt to support more effective organization, direction and utilization of disaster response resources. It appears to be necessary to consider the influence of significantly more information (social, medical, geographical, psychological, political and technological) on agents' behaviour. Therefore the developers claim to need a rich agent model and choose the BDI paradigm where they replace event/goal plans with "situation" plans that capture a broader view of the disaster situation.

As discussed in Section 2.3, the standard BDI paradigm uses a predefined library of plans tagged by their applicability context, and the goal(s) which they seek to achieve. There exist different extensions to the BDI paradigm for equipping agents with fundamental planning capabilities, so they can build their own plans during the simulation: classical planning (e.g. [START_REF] Silva | First principles planning in bdi systems[END_REF]) lets agents synthesise courses of action depending on available domain knowledge; Hierarchical Planning (e.g. [START_REF] Sardina | Hierarchical planning in bdi agent programming languages: a formal approach[END_REF]) supports the generation of high-level plans based on hypothetical reasoning about the relative value of plans.

There are some examples of such extensions being used in the context of multi-agent simulation. Shendarkar et al. [START_REF] Shendarkar | Crowd simulation for emergency response using BDI agents based on immersive virtual reality[END_REF] use agents with a real-time planning capability in a crowd simulation. Schattenberg et al. present JAMES [START_REF] Schattenberg | Planning agents in james[END_REF], a Java-based agent modelling environment for simulation which supports the development of BDI agents with fundamental planning capabilities. The authors claim that "adding planning capabilities to an agent architecture offers the advantage of an explicit knowledge representation combined with a stable and robust decision procedure for the strategic behaviour of the agent". Novak et al. [START_REF] Novák | Simulated multi-robot tactical missions in urban warfare[END_REF] work on autonomous military robots teams for surveillance missions: their first implementation using a reactive hierarchical planner was shown to be inflexible due to its extreme simplicity, and thus later replaced with BDI agents. However these authors noted that BDI agents were too heavy-weight and should be limited to scenarios where complex deliberation was actually needed.

Emotional abilities

Cognitive psychological theories [START_REF] Ortony | The cognitive structure of emotions[END_REF][START_REF] Lazarus | Emotions and adaptation[END_REF] describe emotions in terms of mental attitudes; for example joy is the conjunction of a desire and a belief that it is satisfied. This straightforward connection with the BDI model played in favour of the use of such emotion theories in agents, as noted by [98, p.12]:"one of the appeals of cognitive appraisal models as the basis of computational model is the ease with which appraisal can be tied to the belief, desire and intention (BDI) framework often used in agent systems". Conversely, this easy link between BDI and emotional concepts also constitutes an argument in favour of using BDI agents in simulations where emotional abilities are needed.

The standard BDI framework needs to be extended to integrate emotions (see Section 2.3.4), but it then helps to reason about them and understand their influence on the agents behaviour. Therefore a number of BDI models of emotions have been developed [START_REF] Steunebrink | A formal model of emotions: Integrating qualitative and quantitative aspects[END_REF][START_REF] Adam | The emotions: from psychological theories to logical formalisation and implementation in a BDI agent[END_REF][START_REF] Adam | A logical formalization of the OCC theory of emotions[END_REF], and a number of BDI agents have been endowed with emotions to make them more believable (e.g. Embodied Conversational Agents [START_REF] De Rosis | From greta's mind to her face: modelling the dynamics of affective states in a conversational embodied agent[END_REF][START_REF] Guiraud | The face of emotions: a logical formalization of expressive speech acts[END_REF]) or efficient (e.g. virtual tutors [START_REF] Gratch | Some lessons from emotion psychology for the design of lifelike characters[END_REF]). Such emotional agents also find applications in various types of simulations, for example to model panic in an evacuation scenario [START_REF] Minh | Simulation of the emotion dynamics in a group of agents in an evacuation situation[END_REF] or to model the population possibly irrational behaviour in a bushfire [START_REF] Adam | SWIFT: simulations with intelligence for fire training[END_REF].

In addition to the appraisal process (the way stimuli are appraised to trigger an emotion), the coping process (the way agents deal with their emotions via various strategies) is also very important as it impacts the agents reasoning and behaviour, and as such has already been integrated in some BDI agents [START_REF] Adam | Endowing emotional agents with coping strategies: From emotions to emotional behaviour[END_REF][START_REF] Adam | A bdi emotional reasoning engine for an artificial companion[END_REF]. For example Gratch and Marsella [START_REF] Gratch | A domain-independent framework for modeling emotion[END_REF] have developed a general BDI planning framework integrating appraisal and coping processes to trigger emotions in an agent, based on its interpretation of the world, and to model their influence on its behaviour. This framework called EMA has been applied to military training simulations: human officers interact in a virtual world with virtual humans in order to improve their skills in real-life stressful situations. Complex agent behaviours (in particular emotional behaviours) are needed to add realism and favour immersion, which in turn enhances training.

BDI agents have also been endowed with emotions in simulations aiming at exploring the relationships of emotions with various other concepts. Staller and Petta [START_REF] Staller | Introducing emotions into the computational study of social norms: A first evaluation[END_REF] show that emotions have a bi-directional relationship with norms: emotions can be an incentive to respect norms (e.g. shame or guilt) or to enforce them (e.g. contempt); and norms influence the triggering and regulation of emotions. They have proposed an implementation in TABASCO and aim at reproducing the results of Conte and Castelfranchi [START_REF] Conte | Understanding the functions of norms in social groups through simulation[END_REF]. Nair et al. [START_REF] Nair | The role of emotions in multiagent teamwork[END_REF] have investigated how emotional abilities can improve teamwork in teams composed of either only agents or of both agents and humans; agents simulating human beings have a BDI architecture. Inspired by neurosciences, Bosse [START_REF] Bosse | Modelling collective decision making in groups and crowds: Integrating social contagion and interacting emotions, beliefs and intentions[END_REF] proposes an evacuation simulation where the agent behaviour model is based on beliefs, intentions and emotions and the relationships between these three concepts. Authors also take into account social interactions, and in particular emotional contagion. [START_REF] Jones | Personality, emotions and physiology in a bdi agent architecture: The pep -bdi model[END_REF] have developed a BDI extension called PEP→BDI that enriches the BDI engine with personality, emotions, as well as physiological factors; they apply it to a simulation aimed at training security actors to handle major scale crisis, such as terrorist attacks.

However not all simulations use BDI agents to model emotions, some use very simple mechanisms instead to deal with emotions. For example in [START_REF] Minh | Simulation of the emotion dynamics in a group of agents in an evacuation situation[END_REF], emotion is represented as a single numerical value that increases with perceived danger and influences from others, and decreases with time. Authors claim it is enough to deal with the impact of emotions on evacuation in case of a crisis. But much more complex mechanisms can also be used. For instance Silverman et al. have integrated emotions in their MACES crowd simulation system by using PFMserv [START_REF] Silverman | Crowd simulation incorporating agent psychological models, roles and communication[END_REF], a cognitive architecture claimed to enable more realistic agent behaviour, by providing the agents with a working memory similar to human short-term memory, as well as a long-term memory, both used in their decision cycle.

Although other (more simple or more complex) models of emotions exist, the BDI model brings major benefits in social simulations, because it allows reasoning and interpreting emotions of self and others (reverse appraisal, [START_REF] Rivière | A reasoning module to select ECAs communicative intention[END_REF][START_REF] Adam | Logical modeling of emotions for ambient intelligence[END_REF]), and formalising their influence on behaviour via coping strategies.

Reasoning about norms

Norms are a set of deontic statements (obligation, permission or prohibition) specifying how agents are expected to behave; they are often represented in deontic logic [START_REF] Wright | Norm and Action[END_REF][START_REF] Conte | Understanding the functions of norms in social groups through simulation[END_REF]. Social norms are norms shared by a group of agents; they are a very powerful tool to describe the rules in a group of agents. It is important to consider the fact that these norms can be violated; therefore a mechanism to deal with violations is needed, such as sanctions applied to guilty agents. Each agent has to take norms into account in his decision-making process, i.e. his desires should be balanced by norms. The basic version of BDI does not integrate norms, but some extensions do, e.g. the BOID architecture [START_REF] Broersen | The boid architecture: conflicts between beliefs, obligations, intentions and desires[END_REF][START_REF] Neumann | Norm internalisation in human and artificial intelligence[END_REF] which focuses particularly on conflicts between the four modalities.

Norms are very important in several applications of Multi-Agent Systems (such as Business-to-Business or e-business) but they are particularly essential in the modelling of Socio-Environmental systems [START_REF] Ostrom | A general framework for analyzing sustainability of[END_REF][START_REF] Gaudou | The maelia multi-agent platform for integrated assessment of low-water management issues[END_REF]. Savarimuthu et al. [START_REF] Savarimuthu | A categorization of simulation works on norms[END_REF] draw a quite complete overview of the use of norms in simulation, and categorise simulations w.r.t. their normative mechanisms. In simulations where the norms and their influence on the agents behaviour are of interest, it is important to have a fine-grained model of agents' cognition that allows agents to explicitly reason about norms (see Section 3.4). For example some simulations aim to cognitively explain criminal behaviour [START_REF] Bosse | Towards integration of biological, psychological and social aspects in agent-based simulation of violent offenders[END_REF] and thus use BDI agents.

In [START_REF] Conte | Understanding the functions of norms in social groups through simulation[END_REF], the function of social norms is to control aggression in society. The authors deliberately used very simple agents to study the function of norms at a macro-level. These agents can perform some simple routines (move, eat, attack) to survive in an environment with few food resources; they have a strength that is increased by eating and decreased by movement and attack. The authors compared two strategies to control aggressions in this artificial society: in the normative strategy the agents respect a "finder-keeper" norm, while in the utilitarian strategy they attack for food any agent not stronger than themselves. The simulation showed that the normative strategy was much better to reduce the number of aggressions. Staller and Petta [START_REF] Staller | Introducing emotions into the computational study of social norms: A first evaluation[END_REF] extend [START_REF] Conte | Understanding the functions of norms in social groups through simulation[END_REF] but focus on studying the bi-directional relationship between emotions and social norms: social norms influence the triggering of emotions (e.g. feeling ashamed when violating a norm) and their regulation and expression (for example one should not laugh at a funeral); and emotions in turn influence normative behaviour (for example shame drives people to respect norms, while contempt drives them to apply sanctions to enforce norms). Due to their focus on the micro-level and its interactions with the macro-level, the authors cannot content themselves with the very simple agent model used in [START_REF] Conte | Understanding the functions of norms in social groups through simulation[END_REF]. Instead they use the TABASCO three-layer agent architecture, and a BDI architecture called JAM for the reasoning part of their agents.

BDI architectures seem very adapted to model two mechanisms: the internalization of norms, and the influence of norms on the decision-making process. Indeed, the explicit representation of norms allows an agent to reason about them and make autonomous decisions regarding their internalisation (instead of automatically accepting all norms); based on its desires and beliefs, the agent can thus choose to violate norms (such as in [START_REF] Castelfranchi | Deliberative normative agents: Principles and architecture[END_REF]).

One of the main critics against BDI architecture regarding norms is that such architectures use a priori existing norms (implemented offline). The emergence and recognition of norms at runtime can be handled with other architectures such as EMIL [START_REF] Campenn | Normal = Normative? The Role of Intelligent Agents in Norm Innovation[END_REF]; but this problem has only been marginally and very recently tackled in the BDI literature. [START_REF] Criado | A BDI Architecture for Normative Decision Making[END_REF][START_REF] Criado | Towards a normative bdi architecture for norm compliance[END_REF] extend a graded BDI architecture to allow autonomous (and agent-dependent, based on their preferences, contrarily to EMIL) norm acceptance.

To conclude, BDI architectures offers a powerful tool to reason on norms and to integrate them in the decision process, but are more adapted to simulations with initially established norms. Therefore, the emergence of informal social norms in groups of agents would be harder to tackle with a BDI architecture. However, this restriction is not so strong since BDI architectures are still applicable to the large number of simulations where laws and regulations are established by a superior institution.

Decision-making

First of all, we can notice that the decision-making process is often not key in BDI frameworks and architectures. Norling [START_REF] Norling | Folk psychology for human modeling: extending the BDI paradigm[END_REF] notices that BDI theory assumes that agents make utility-based rational decisions, optimising the expected utility of the selected action, and that in implementations (e.g. JACK) agents usually choose the first applicable plan regardless of its expected utility. But Norling claims that both strategies are not representative of human decision-making, and proposes to extend BDI with a different decision strategy: recognition-primed decision. This strategy is typically used by experts who spend most of the time finely analysing the situation, before making an almost automatic decision based on this situation. Some works show that the BDI paradigms eases the modelling of agents with a more realistic decision-making process. Lui et al. [START_REF] Lui | An architecture to support autonomous command agents for onesaf testbed simulations[END_REF] model military commanders in land operations scenarios with a BDI approach, which they claim to provide the agents with the ability to demonstrate human-like decision-making. Noroozian et al. [START_REF] Noroozian | Towards simulating heterogeneous drivers with cognitive agents[END_REF] use BDI agents to model individual drivers with heterogeneous driving styles for a more realistic traffic model. Indeed, the BDI paradigm makes it easier to design agents with various decision criteria that can react differently to the same stimulus.

In addition, the BDI framework makes it easy to integrate and explore the influence of several cognitive factors on the decision-making process. Bosse et al. [START_REF] Bosse | Integrating rational choice and subjective biological and psychological factors in criminal behaviour models[END_REF][START_REF] Bosse | Towards integration of biological, psychological and social aspects in agent-based simulation of violent offenders[END_REF] present a BDI modelling approach to criminal decision making in a violent psychopath agent. Their model is based on criminology theories that define opportunities for a crime. Desires are generated from biological, cognitive and emotional factors; beliefs in opportunities are generated from the observation of the social environment; their combination leads to intentions and actions to satisfy these desires. The implementation is done with the authors' LEADSTO language that allows them to combine quantitative (physiological hormones levels) and qualitative (mental attitudes) concepts in a unified model. In a socio-environmental application, Taillandier et al. [START_REF] Taillandier | A new BDI agent architecture based on the belief theory. Application to the modelling of cropping plan decision-making[END_REF] build a model to deal with farmers' decision-making about cropping plans, farmers behaviours being described with a BDI architecture. To this purpose, they have used a multi-criteria decision-making process, considering the desires as the criteria that drive the decision. Finally, Swarup et al. [START_REF] Swarup | Computational epidemiology as a challenge domain for multiagent systems[END_REF] argue that a BDI model of agents can be fully relevant to understand and model seemingly irrational human decision making, such as H1N1 vaccination refusal, and its influence on the dynamics of an epidemic.

Summary

To summarise, the BDI model is quite agnostic in terms of the decision-making approach, and many algorithms can be used. Nevertheless, modellers can benefit from the high-level BDI concepts to design a much finer decision-making process than the simple economic maximizer agent. The BDI framework also allows to model individualised, possibly irrational, decision-making, where each agent makes its decision based on its own motivations. This is essential to simulate realistic human societies for various goals in various fields of application and for various goals. Table 1 summarises the importance and benefits of using a BDI architecture for each of the characteristics above.

Agent

Influence of the field of application

Macal and North in their tutorial on ABMS [START_REF] Macal | Tutorial on agent-based modeling and simulation[END_REF] list a number of fields in which there are practical applications of ABMS. We examine each of these fields as well as some other more recent ones, and discuss the appropriateness and benefits of a BDI approach in each case.

In business and organisations simulations: most manufacturing applications involve clear rules and lists of tasks, so a goal-based approach like BDI is not the most appropriate; on the contrary, BDI offers modellers of consumer markets a straightforward mapping between their understanding of the purchasing decision-making process and the BDI concepts (see for example [START_REF] Baptista | An agent-based model of consumer behaviour based on the BDI architecture and neoclassical theory[END_REF]); finally in supply chains and insurance simulations, depending on the complexity and uncertainty in the models, BDI may also help as it allows a more complex but still transparent model.

In economics, artificial financial markets and trade networks models are presently often over simplified by assuming perfectly rational agents. By using BDI agents, emotions and norms, a more reasonable assumption of the human reasoning process could be captured [START_REF] Wan | Decision-making algorithm of an agent's internal behavior facing artificial market[END_REF].

In infrastructure simulations, the participants of electric power markets or hydrogen economies need to "perform diverse tasks using specialized decision rules" [START_REF] Macal | Tutorial on agent-based modeling and simulation[END_REF], which could be achieved by specifying a plan library and allowing the BDI paradigm to capture the complexity. Nowadays with power grids, the electric power market model needs to integrate micro-behaviours of inhabitants to be more accurate; BDI agents are relevant to simulate "social, perception, cognitive and psychological elements to generate inhabitants' behaviour" [START_REF] Kashif | Agent based framework to simulate inhabitants' behaviour in domestic settings for energy management[END_REF], based on a reasoning about potential consequences of their actions to find the most energy-efficient behaviour.

In transportation simulations, BDI is ideal for modelling humans who have many different ways of using the transportation systems, each with its own benefits. The self-explanatory nature of BDI agents also allows the modeller to understand why one choice was made over another. It can also help to model individual drivers with heterogeneous driving styles [START_REF] Noroozian | Towards simulating heterogeneous drivers with cognitive agents[END_REF] for a more realistic traffic model.

In crowds simulations, in particular human movement and evacuation modelling, BDI offers more precise and realistic modelling of human behaviour, in order to enrich the crowd simulation [START_REF] Shendarkar | Crowd simulation for emergency response using BDI agents based on immersive virtual reality[END_REF][START_REF] Cho | Bdi model-based crowd simulation[END_REF]. For instance Park [START_REF] Park | Modeling social group interactions for realistic crowd behaviors[END_REF] made a crowd simulation more realistic by introducing social interactions between the agents.

More generally, crisis management needs tools for decision-support for the stakeholders, training of the risk managers, and education and awareness of the population. Such tools require realistic models for better immersion and for valid results. For example Paquet et al. [START_REF] Paquet | DAMAS-Rescue Description Paper[END_REF] model rescue workers after a natural crisis in DamasRescue using BDI agents; Buford et al. [START_REF] Buford | D-aesop: a situation aware bdi agent system for disaster situation[END_REF] develop a Situation-Aware BDI Agent System for Disaster Situation Management.

In epidemiology, Swarup et al. present the BDI model as very good for understanding seemingly irrational human decision making such as H1N1 vaccination refusal, and their influence on the dynamics of an epidemic [START_REF] Swarup | Computational epidemiology as a challenge domain for multiagent systems[END_REF].

In society and culture: when simulating the fall of ancient civilisations such as in [START_REF] Axtell | Population growth and collapse in a multiagent model of the Kayenta Anasazi in Long House Valley[END_REF][START_REF] Heckbert | Mayasim: an agent-based model of the ancient maya social-ecological system[END_REF], BDI agents are irrelevant for several reasons: the time scale of the simulation (several decades); the lack of precise behaviour data; and the focus on external influences (environment, trading) rather than individual decision making. Nevertheless, BDI agents could be useful when simulating shorter and more recent historical periods where more data is available (e.g. the 1926 flood management in Hanoi [START_REF] Gasmi | Reproducing and exploring past events using agent-based geo-historical models[END_REF]). The concepts of norms, as well as the heterogeneity of behaviours among people in an organisation, also make BDI appropriate for modelling organisational networks.

In military simulations, since BDI reflects the way people think they think (meta-reasoning), it is perfect for simulating people in a participatory command and control simulation [START_REF] Lui | An architecture to support autonomous command agents for onesaf testbed simulations[END_REF][START_REF] Gratch | A domain-independent framework for modeling emotion[END_REF]. The BDI approach has also been used in this field for the realistic decision-making it brings. Karim et al. propose an autonomous drone pilot with BDI reasoning [START_REF] Karim | Experiences with the design and implementation of an agent-based autonomous uav controller[END_REF]; Novak et al. use BDI reasoning for autonomous surveillance robots in urban warfare, because it is more flexible than planning and more practical than tele-operation [START_REF] Novák | Simulated multi-robot tactical missions in urban warfare[END_REF].

In criminology, BDI agents are used as they can manage various cognitive concepts in a unified architecture. In [START_REF] Bosse | Towards integration of biological, psychological and social aspects in agent-based simulation of violent offenders[END_REF][START_REF] Bosse | Integrating rational choice and subjective biological and psychological factors in criminal behaviour models[END_REF], Bosse et al. investigate the influence of psychological (emotions) as well as social and biological aspects on criminal behaviour; their model merges qualitative BDI aspects and quantitative biological aspects.

In ecology, the lack of a decision making process in the agents being modelled suggests that BDI is not appropriate. In biology: for animal group behaviour, BDI may be appropriate, depending on the animal being modelled and its cognitive abilities; for cells or sub-cellular molecules simulations, the simple nature of the behaviours makes BDI unsuitable.

Intuitively, BDI agents are more adapted in fields where the agents modelled are humans and the decision process is well known, and less adapted to the modelling of simpler non human entities or when there is no decision making process.

Influence of the goal of the simulation

ABMS consists of two steps: first, create an agent-based model of a target theory or from observed data; second, build and run a simulation of this model. Axelrod [START_REF] Axelrod | Advancing the art of simulation in the social sciences[END_REF] admits that depending on the goal of an agent-based model one should emphasize different features. Concretely, when the goal is to enrich our understanding of a phenomenon, the model has to stay simple to facilitate analysis of the results: "when a surprising result occurs, it is very helpful to be confident that one can understand everything that went into the model"; it also allows easier replication or extension of a model. On the other hand, when the goal is to faithfully reproduce a phenomenon, and realism and accuracy are important, the model might need to be more complicated: "there are other uses of computer simulation in which the faithful reproduction of a particular setting is important [...]. For this purpose, the assumptions that go into the model may need to be quite complicated". Once one has designed a model, appropriately balancing accuracy and simplicity, the next step is to run a simulation, which can be used to reach various goals. Indeed Axelrod [START_REF] Axelrod | Advancing the art of simulation in the social sciences[END_REF] lists seven purposes of simulations. In the following paragraphs we present each of them, provide examples of simulations having this goal, and discuss if BDI agents are appropriate to reach it or not, and why.

Prediction

Simulations can be used to predict potential outputs resulting from known complex inputs and assumptions about mechanisms.

For example Gunderson et al. [START_REF] Gunderson | Using a multi-agent model to predict both physical and cyber criminal activity[END_REF] use data mining to create a simulation from a large quantity of statistical data available about crimes, and then use it to predict criminal behaviour. The authors are not interested here in the details of the agents cognition, but in the global crime pattern at the scale of a city, and therefore do not need BDI agents (nor have the micro-level data needed to design a BDI model, plus it would add unneeded complexity and scalability problems). In a different field, Baptista et al. [START_REF] Baptista | An agent-based model of consumer behaviour based on the BDI architecture and neoclassical theory[END_REF] model consumers with a realistic underlying BDI agent model, which they claim to be valuable for predicting market trends.

Simulations that can predict the behaviour of the population in response to various policies are also a powerful tool for decision-support, on condition that the underlying model is as faithful as possible in order to lay valid results. A BDI architecture of agents representing humans thus ensures the right level of realism. Indeed, Swarup et al. in their prospective paper [START_REF] Swarup | Computational epidemiology as a challenge domain for multiagent systems[END_REF] claim that the BDI model is very well-adapted to model (possibly irrational) human decision making, such as H1N1 vaccination refusal, when simulating and predicting the dynamics of an epidemic.

Such decision-support systems can be used to assist policy makers in various fields. For example, Sokolova et al. [START_REF] Sokolova | An agent-based decision support system for ecological-medical situation analysis[END_REF] present a decision-support system that analyses pollution indicators and evaluates the impact of exposure on the health of the population, modelled as BDI agents; their system allows ecological policy-makers to explore what-if scenarios, giving them evidence to support decision-making. Shendarkar et al. [START_REF] Shendarkar | Crowd simulation for emergency response using BDI agents based on immersive virtual reality[END_REF] propose a crowd model with BDI agents to simulate evacuation in case of a terrorist attack, in order to determine where it is more useful to place policemen to improve evacuation.

To conclude, the BDI architecture is a good candidate in simulations aimed at predicting future behaviour, because this requires the behaviour of agents to be as realistic as possible, adaptable to a large number of situations. However it can only be used if there is enough detailed data available: this will be discussed in Section 3.5.1.

Performance of a task

Axelrod [12, p.16] claims that "to the extent that artificial intelligence techniques mimic the way humans deal with these tasks, the artificial intelligence method can be thought of as simulation of human perception, decision making, or social interaction". We therefore sort in this category Artificial Intelligence research that attempts to simulate human processes into agents to improve their efficiency at these tasks.

For example the Virtual Humans community aims at creating artificial humans endowed with a wide range of humanlike abilities (including face-to-face multimodal interaction, emotions, personality...) [START_REF] Gratch | Creating interactive virtual humans: Some assembly required[END_REF][START_REF] Swartout | Toward virtual humans[END_REF]. Agents endowed with good (human-like) communication skills are useful as interface agents or as virtual companions [START_REF] Wilks | Close Engagements With Artificial Companions: Key Social, Psychological, Ethical and Design Issues. Natural Language Processing[END_REF].

When virtual humans aim at a more realistic or complex behaviour, they usually rely on a BDI architecture, possibly extended with additional required features. For example BDI agents with human-like negotiation skills have been used as mentors in role-playing games to train humans to improve their own negotiation skills in stressful situations [START_REF] Traum | Negotiation over tasks in hybrid human-agent teams for simulation-based training[END_REF]. Another BDI agent endowed with emotions could express realistic emotions or to detect the user's ones in Ambient Intelligence scenarios [START_REF] Adam | Logical modeling of emotions for ambient intelligence[END_REF]. Emotions and associated coping strategies were also used as human-inspired heuristics in a BDI reasoner for efficient robot behaviour [START_REF] Adam | A bdi emotional reasoning engine for an artificial companion[END_REF].

BDI agents with human-like emotional abilities have also been shown to perform better teamwork. Nair et al. [START_REF] Nair | The role of emotions in multiagent teamwork[END_REF] use a BDI model of coordination, combined with POMDP to handle uncertainty. In pure agents teams, their BDI emotional agents have a more realistic behaviour, since humans are known to use emotions for goal prioritisation and action selection. In mixed humans-agents teams, they are better team-mates: awareness of their human partners' emotions improves communication with them, and expression of their emotions contributes to building a relationship with them.

Peinado et al. [START_REF] Peinado | Revisiting character-based affective storytelling under a narrative bdi framework[END_REF] present BDI as a "promising tool for modelling sophisticated characters in Interactive Storytelling". They mention in particular that it allows to create richer characters with their own beliefs and motivations causing their behaviour [START_REF] Karim | Experiences with the design and implementation of an agent-based autonomous uav controller[END_REF] use BDI agents to design an autonomous pilot for military Unmanned Aerial Vehicles ; they find that the BDI paradigm provides a more modular design, a high-level of abstraction, and improves scalability by avoiding hard-coded coordination.

So BDI agents seem to provide a good paradigm to realistically model human behaviour in agents aimed either at behaving in a believable, human-like way, or at performing tasks more efficiently by reproducing the human way of performing them. Indeed BDI provides a high-level description of such behaviour, so it is the perfect level of abstraction to efficiently capture psychological theories of human behaviours. Such agents endowed with human-like abilities find a very wide variety of applications, from virtual companions or storytellers to military UAV pilots.

Training

These simulations aim to "train people by providing a reasonably accurate and dynamic interactive representation of a given environment" [START_REF] Axelrod | Advancing the art of simulation in the social sciences[END_REF]. Axelrod then notices that simulations having such a goal need accuracy and realism rather than simplicity. Indeed lots of training simulations do use complex BDI agents, often extended with additional abilities such as emotions or negotiation skills.

A lot of works have used BDI agents in serious games for training, because they provide a human-like decision-making process described in terms of folk psychology. Baptista et al. [START_REF] Baptista | An agent-based model of consumer behaviour based on the BDI architecture and neoclassical theory[END_REF] thus model consumers with a realistic underlying BDI agent model, and claim it can be useful for training management skills of business students. Another application is the use of virtual humans as virtual patients to train medical professionals. Rizzo et al. [START_REF] Rizzo | Intelligent virtual patients for training clinical skills[END_REF] have used a virtual human with a complex BDI architecture to create different virtual patients that provide realistic representations of mental problems to be diagnosed or discussed. Such virtual patients are useful to improve doctors' interview and diagnostic skills for a difficult subject through practice.

The same virtual human has also been used in a variety of other simulations for the military. For example it has been used in a war scenario, the Mission Rehearsal Exercise, to train military task-negotiation [START_REF] Traum | Negotiation over tasks in hybrid human-agent teams for simulation-based training[END_REF] and decision-making [START_REF] Gratch | A domain-independent framework for modeling emotion[END_REF] skills. Traum et al. then enriched it with conversation strategies [START_REF] Traum | Fight, flight, or negotiate: Believable strategies for conversing under crisis[END_REF] to create a virtual negotiator that can play the role of a mentor to train military negotiation skills in crisis situations; the trainee learns principles of negotiation by interacting with the agent using them to play the opposing party. Many other military applications can also been found. [START_REF] Mcilroy | Air combat tactics implementation in the smart whole air mission model[END_REF] developed the Smart Whole AiR Mission Model that splits the physics of combat and pilot reasoning into two separate entities which are modelled, validated, and implemented in completely different ways; fighter pilots have a BDI architecture based on the dMARS [START_REF] D'inverno | The dmars architecture: A specification of the distributed multi-agent reasoning system[END_REF] realisation of the Procedural Reasoning System. Lui et al. [START_REF] Lui | An architecture to support autonomous command agents for onesaf testbed simulations[END_REF] use Jack Teams, a BDI team modelling framework, to model military commanders in land operations scenarios.

BDI agents have also been used because of their adaptability to a changing environment. Oulhaci et al. [START_REF] Oulhaci | A multi-agent architecture for collaborative serious game applied to crisis management training: Improving adaptability of non played characters[END_REF] designed a serious game for training in the field of crisis management, where Non-Played Characters have a complex behaviour, either behaving as expected or intentionally erroneously. These NPC are developed as BDI agents so that they can adapt to events of the world and to social interactions with and actions of other players (in particular the learner). Cho et al. [START_REF] Cho | Bdi model-based crowd simulation[END_REF] used BDI agents to simulate crowds in virtual worlds, claiming that BDI increases their adaptability to the dynamic environment, and therefore their realism. They present a prototype game with non-played characters extinguishing a forest fire, implemented with the SourceEngine 3D game engine.

To conclude, simulations for training require a realistic immersive environment to be efficient. In particular, the behaviour of autonomous agents populating the virtual world (non-player characters) must be human-like, making BDI agents a very good choice, for their descriptivity of the human-like decision-making process, and their adaptability to a dynamic environment.

Entertainment

This goal is quite similar to training but the virtual worlds can be imaginary ones. There has been work on integrating BDI agents into games in order to provide the players with a richer, more enjoyable experience.

A number of works are dedicated to the integration of autonomous agents (bots) in Unreal Tournament. Small [START_REF] Small | Agent smith: a real-time game-playing agent for interactive dynamic games[END_REF] uses a simple rule-based agent called Steve, that can quickly decide its action (e.g. move to healing source) based on high-level conditions (e.g. weak energy) in a dynamic environment. To be able to compete with human players, Steve mimicks their learning process through an evolutionary algorithm: its strategies are evolved (and improved) over time. However it was still unable to consistently beat human challengers. Hindriks et al. [START_REF] Hindriks | Unreal goal bots. connecting agents to complex dynamic environments[END_REF] wanted to find alternatives to such reactive bots. They argue that using BDI agents instead of scripts or finite-state machines (often used in real-time games) might result in more human-like and believable behaviour, thus making the game more realistic. They add that when one gets data from observing actual game players, it is easier to specify it in terms of beliefs, desires and intentions and import it into BDI agents, than to translate it into finite-state machines. They thus tried to interface the GOAL agent programming language with Unreal Tournament 2004. Such an interface raises two main issues: finding the right level of abstraction of reasoning (the BDI agent should not have to deal with low-level details, while keeping enough control), and avoiding cognitive overload to achieve real-time responsiveness (the agent should not receive too many percepts, but should have enough information, even if possibly incomplete and uncertain, to make decisions). One of their motivations was to test agent programming platforms, in order to eventually make them more effectively applicable. They indeed noticed that BDI agent technology is not scalable yet, and they observed bad performance when increasing the number of agents.

Another try at integrating BDI agents into Unreal Tournament is described in [START_REF] Urlings | A future framework for interfacing bdi agents in a real-time teaming environment[END_REF]. The authors are actually interested in humans-agents teams in the area of unmanned vehicles. They propose a framework to model these and then use the Unreal Tournament game as a real-time simulation environment to demonstrate this framework. Their agents are implemented with a BDI architecture using JACK. BDI agents have also been used in other games: Norling [START_REF] Norling | Capturing the quake player: using a bdi agent to model human behaviour[END_REF] describes a BDI Quake player; the main character of the Black and White game [START_REF] Molyneux | Postmortem: Lionhead studios' black and white[END_REF] is also a BDI-based agent. In another attempt to make video games more immersive and realistic (and not only graphically so), Palazzo et al. [START_REF] Palazzo | Spyke3d: A new computer games oriented bdi agent framework[END_REF] provide the Spyke3D framework, using BDI agents to facilitate the creation of more human-like Non-Player-Characters with plausible logical behaviours.

The works described above all concern video games, but BDI can also be used to model characters in interactive stories and endow them with a richer causal behaviour based on their beliefs and motivations [START_REF] Peinado | Revisiting character-based affective storytelling under a narrative bdi framework[END_REF].

So BDI agents are an excellent tool to create believable characters that make games or stories more realistic and immersive. Unfortunately there are performance issues when the number of agents increases, which curbs their wider use in games so far, when designers usually prefer to improve graphics and rely and simple but fast scripted behaviours. We will discuss solutions to the scalability problem in Section 3.5.2.

Education

According to Axelrod, "the main use of simulation in education is to allow the users to learn relationships and principles for themselves" [START_REF] Axelrod | Advancing the art of simulation in the social sciences[END_REF]. So this goal is similar to training (and entertainment), but the virtual world does not need to be as accurate (Axelrod says they "need not be rich enough to suggest a complete real or imaginary world"), as long as it lets the student experiment and learn by trial and error. For example in the SmartEnergy project, Kashif et al. [START_REF] Kashif | Agent based framework to simulate inhabitants' behaviour in domestic settings for energy management[END_REF] propose a serious game aimed at raising awareness about energy saving; the realism is not key here as the main requirement is to give the users an insight of the effects of their actions.

Nevertheless to improve the users' learning, it is often useful to provide them with an artificial tutor inhabiting this virtual world, and able to explain to them why they did wrong or right. Therefore Herbers et al. [START_REF] Harbers | Explaining simulations through self-explaining agents[END_REF][START_REF] Harbers | Self-explaining agents[END_REF] use BDI agents that can provide such explanations in terms of intuitive concepts explicitly represented in the BDI architecture: their goals and beliefs (see Section 3.1.2). For example, Elliott et al. [START_REF] Elliott | Lifelike pedagogical agents and affective computing: An exploratory synthesis[END_REF] study the integration of emotions in a virtual tutor called Steve that inhabits a 3D world and teaches students to operate a high-pressure air compressor; Steve has also been used in a training simulation with an army peacekeeping scenario to prepare officers to deal with difficult situations [START_REF] Rickel | Steve goes to bosnia: Towards a new generation of virtual humans for interactive experiences[END_REF].

To conclude, in education-oriented simulations realism of the virtual world and of its population is not key, so it is not necessary to use a BDI architecture for all agents. Nevertheless the BDI architecture is really useful for some specific agents that need to be able to explain their behaviour or the user's errors, such as some NPC or an artificial tutor, in order to improve learning.

Proof

Another possible purpose of simulations identified by Axelrod is to provide an existence proof. For instance Conway's Game of Life [START_REF] Gardner | The fantastic combinations of john conway's new solitaire games[END_REF] is a game based on a set of simple rules governing the birth and death of cells, which can be used to prove or disprove conjectures such as "a finite population cannot grow infinitely large".

The literature provides few examples of agent-based models aimed at existence proofs. One famous example however is Schelling's agent-based segregation model [START_REF] Schelling | Dynamic Models of Segregation[END_REF]. Very simple rules govern the agents' movement to other neighbourhoods based on their level of happiness with their neighbours. Just a small desire to live near people with similar traits can quickly lead to large clusters of homogeneous neighbourhoods. This model is considered to provide an existence proof that segregation does not require the people involved to be highly intolerant, but only mildly so.

The line between exploration and proof is difficult to draw. Axtell et al. [START_REF] Axtell | Population growth and collapse in a multiagent model of the Kayenta Anasazi in Long House Valley[END_REF] provide a multi-agent model of the exodus of the Kayenta Anasazi (modelled as relatively simple reactive agents) from Long House Valley, as a means to explore its possible causes. Whilst it provides a strong suggestion that the inability of the land to sustain the population was not the only cause of the exodus (by showing that some subset of the population could have been sustained) it falls short of being considered an existence proof that sociocultural factors had some influence. It is not clear whether this is because this relatively simple model is in fact too complex, or indeed too basic to support such reasoning.

Sun and Naveh [START_REF] Sun | Social institution, cognition, and survival: a cognitive-social simulation[END_REF] argue that socio-cognitive simulation is essential to prove functionalist hypothesis, that justify social phenomenons by the function they have in society, i.e. that explain a cause by its effect. Such hypothesis are usually hard to verify, but social simulation allows to verify or prove them through experimentation. However the rudimentary cognition assumed in agents used in these simulations does reduce their realism and hence their applicability, and also prevents from proving links between the micro (cognitive) and macro (social) levels of the simulation. The authors therefore argue for the use of a specific cognitive architecture called CLARION; we claim that BDI agents would achieve the same purpose (as discussed above in Section 2.2).

To conclude, depending on the hypotheses that the simulation aims at proving, the complexity required from the agents involved is not the same. Hypotheses about the survival of a cell population, or the geographical movements of a human population, can be proved with very simple agents; in contrast, more complex hypotheses about the links between individual cognition and social organisation do require more complex agent models, with BDI being one option.

Discovery

The last purpose of simulations identified in Axelrod's classification is knowledge discovery. Axelrod claims that social scientists have successfully "discover[ed] important relationships and principles from very simple models" and argues that "the simpler the model, the easier it may be to discover and understand the subtle effects of its hypothesised mechanisms" [START_REF] Axelrod | Advancing the art of simulation in the social sciences[END_REF].

Sun has tried to discover relationships and interactions between the micro-level (cognition of agents) and macro-level (dynamics of society) [START_REF] Sun | Desiderata for cognitive architectures[END_REF], for example in the simulation of social groups depending on the food management strategy [START_REF] Sun | Social institution, cognition, and survival: a cognitive-social simulation[END_REF]. He obviously needed a fine-grained model of agents, and used his CLARION cognitive architecture. Heinze et al. [START_REF] Heinze | Interchanging agents and humans in military simulation[END_REF] model airborne early warning and control crew with BDI agents in a simulation used for the exploration, evaluation and development of tactics and procedures, in order to advise the Australian Defence Force in the conduct of maritime surveillance and patrol.

ABMS have also been used recently as a tool for public policy analysis, as it provides a means to discover possible sources of policy resistance. Policy resistance is the tendency for policy interventions to be defeated by unpredicted effects. For example, making public fuel-efficient transportation more affordable to the general public may reduce carbon emissions around the city, but the money saved due to reduced cost of local road travel might be spent on leisure, leading to an increase in air travel, resulting in no overall reduction in carbon emissions. Such effects are generally not intuited because the system is constantly changing, adaptive and evolving, with tightly coupled actors and nonlinear effects [START_REF] Sterman | Learning from evidence in a complex world[END_REF] -factors which are well addressed by agent-based modelling.

For example the Electricity Market Complex Adaptive System multi-agent simulation has been used to explore the restructuring and deregulation of the electric power market in Illinois [START_REF] Cirillo | Evaluating the potential impact of transmission constraints on the operation of a competitive electricity market in illinois[END_REF], and provided information on this policy issue "that would otherwise have not been available using any other modelling approach" [START_REF] Macal | Tutorial on agent-based modeling and simulation[END_REF]. There are also many examples of agent-based simulations being used to study climate policy (e.g. [START_REF] Moss | Agent-based integrated assessment modelling: the example of climate change[END_REF][START_REF] Posada | Emissions permits auctions: An agent based model analysis[END_REF]).

Whilst policy development is an emerging field of application for ABMS, the tight link between policy impact and human behaviour strongly suggests a need for realistic, rich models of the latter, as supported by the BDI paradigm. In the context of using agent simulation for exploring economic policy, Farmer and Foley [START_REF] Farmer | The economy needs agent-based modelling[END_REF] suggest that "the major challenge lies in specifying how the agents behave and, in particular, in choosing the rules they use to make decisions".

Conclusion

To summarise, we showed in this section that the BDI architecture can bring benefits to simulations whatever their goal, but more particularly so for some goals. Simulations for prediction needs a more realistic underlying model for all agents (with BDI being a candidate of choice for human agents) to accurately support decision and to lay valid results. When simulations aim at imitating human processes for performing a task, BDI provides the right abstraction level to capture the modeled reasoning. Simulations aiming at training, entertainment or education benefit from realistic and adaptive agents, that enhance immersion and engagement, and increase efficiency by providing explanations; BDI agents are already widely used in such simulations. Finally, simulations for proof and discovery should remain simple at first, but BDI agents can then be useful to prove social phenomena involving individual cognition of agents. Table 2 summarises the interest, benefits and drawbacks of using the BDI approach depending on the simulation purpose. Independently of the goal of the simulation, the choice of a BDI architecture for the agents also depends on other factors, such as the required level of granularity or observation scale (does the simulation focus on the micro or macro level?), and the (type and quantity of) data available to design the model. These will be discussed below.

Simulation

Influence of observation level

In this section we argue that the desired level of observation, i.e. the focus on micro vs macro level phenomena, influences the choice of the agents architecture. Models in which the agent is the center of interest seem to be more likely to use a BDI architecture, since the modeller tends to endow the agents with as realistic a behaviour as possible (for instance for NPC in video games [START_REF] Norling | Capturing the quake player: using a bdi agent to model human behaviour[END_REF], as discussed in Section 3.3.4). This is particularly true in models designed to investigate in details the effects of psychological or physiological factors on individual cognition (for instance in criminal behaviour [START_REF] Bosse | Integrating rational choice and subjective biological and psychological factors in criminal behaviour models[END_REF][START_REF] Bosse | Cognitive and social simulation of criminal behaviour: the intermittent explosive disorder case[END_REF] as discussed in Section 3.2).

On the contrary, simulations that are focused on the dynamics of a large population (e.g. crowds simulations or population dynamics) do not need such a fine granularity, and thus generally use simple agents to reduce computing time and enhance performances [START_REF] Wolfe | To bdi, or not to bdi: design choices in an agent-based traffic flow management simulation[END_REF]. For example a lot of crowd simulations use distributed behaviours producing a global flocking behaviour [START_REF] Reynolds | Flocks, herds, and schools: A distributed behavior model[END_REF]. However as soon as modelers want to simulate a more realistic crowd, for example to improve immersion of the human player in a virtual environment, they try to use a BDI architecture [START_REF] Shendarkar | Crowd simulation for emergency response using BDI agents based on immersive virtual reality[END_REF][START_REF] Silverman | Crowd simulation incorporating agent psychological models, roles and communication[END_REF][START_REF] Cho | Bdi model-based crowd simulation[END_REF]. The focus of observation has shifted from a global point of view on the overall behaviour of the crowd (e.g. flocking) to a local point of view on the behaviour of individual agents composing the crowd.

For example in criminology, some authors focus on global crime patterns at the scale of a city and use statistical data in a simple agent behaviour model [START_REF] Gunderson | Using a multi-agent model to predict both physical and cyber criminal activity[END_REF], while others investigate the effects of psychology and physiology on individual criminal behaviour with a more complex agent model merging qualitative BDI and quantitative biological aspects [START_REF] Bosse | Cognitive and social simulation of criminal behaviour: the intermittent explosive disorder case[END_REF][START_REF] Bosse | Towards integration of biological, psychological and social aspects in agent-based simulation of violent offenders[END_REF]. We could also cite traffic simulation, where very simple models of the traffic flow exist [START_REF] Nagel | A cellular automaton model for freeway traffic[END_REF], but some authors recently used a BDI model of drivers to focus on their individual driving styles [START_REF] Noroozian | Towards simulating heterogeneous drivers with cognitive agents[END_REF].

We can conclude that the level of observation of the simulation has a great influence on the choice of an agent architecture: if the focus is on observing the global behaviour of the group of agents, then simple agents are sufficient and should be used to optimise performances of the simulator; but when the individual agents in the group become the focus, a more fine-grained and realistic approach has to be favoured, and the BDI model is more appropriate. However such an approach also requires more fine-grained data to elaborate the model; this is discussed below in Section 3.5.1.

Features of a simulation that deter from using BDI agents

In this section we discuss the factors traditionally cited as limits of the BDI architecture and hindering its wider use in social simulations, along with possible solutions to overcome these limits. We discuss the lack of fine-grained data to design the model (Section 3.5.1), the scalability issue (Section 3.5.2), the limited expressivity (Section 3.5.3) and the absence of learning (Section 3.5.4).

Influence of the availability of data

To build an model as realistic and useful as possible, the data available about the reference system has a huge importance, and can even influence the choice of the agent architecture. Indeed, if the only data available describes the behaviour of social groups (e.g. family statistics) the modeller does not have enough precise information to give a very rich architecture to the agents representing the individual members of these groups [START_REF] Gil-Quijano | Mechanisms of automated formation and evolution of social-groups : A multi-agent system to model the intra-urban mobilities of bogota city[END_REF]. For instance [START_REF] Gunderson | Using a multi-agent model to predict both physical and cyber criminal activity[END_REF] use data mining to deal with the large quantity of statistical data available about crimes; this lets them model the global crime pattern at the scale of a city, but not the details of the agents cognition. Therefore, when the goal of the simulation (as discussed above in Section 3.3) requires realism, in particular when the simulation is used for prediction, a great deal of attention has to be paid to the collection and processing of precise data, and thus the use of efficient and dedicated techniques and tools.

It is generally much easier to collect objective and measurable data (such as age, income...), which is sufficient to build a simple reactive agent model, than to gather subjective information explaining behaviour in terms of (not observable) mental states. But since this kind of subjective information is needed to build rich agent models such as BDI agents, another approach is to use participatory modelling: put a human in a virtual world to capture his behaviour, and then reproduce it in the agents of the simulation [START_REF] Shendarkar | Crowd simulation for emergency response using bdi agents based on immersive virtual reality[END_REF][START_REF] Shendarkar | Crowd simulation for emergency response using BDI agents based on immersive virtual reality[END_REF]. For instance, Silverman et al. collected attributes values governing BDI characteristics by placing human beings into a Virtual Reality environment, to build a BDI model of the crowd behaviour after a terrorist attack [START_REF] Shendarkar | Crowd simulation for emergency response using BDI agents based on immersive virtual reality[END_REF]. The MAELIA project [START_REF] Gaudou | The maelia multi-agent platform for integrated assessment of low-water management issues[END_REF] investigates policies for water resource management; as the water used in the studied basin is mainly influenced by farming practices, the modellers interviewed farmers about how they made their decisions regarding cropping.

So the BDI model needs precise data about individual behaviours, which is harder to collect. But the BDI model also offers advantages, in particular its descriptivity. Indeed, Edmonds et al. [47, p.4] claim that descriptive models (in particular agent-based) can help modellers to exploit more of the available data, in particular qualitative information such as anecdotal or commonsense evidence, because of the straightforward correspondence with concepts used in the model; it thus combines well with participatory modelling. Norling also notices that it is easier for experts being modelled to provide information in an intuitive language: "people have a tendency to explain their actions in terms of their intentions, which in turn are explained in terms of their goals and beliefs" [102, p.2]. She thus argues for the use of a descriptive approach such as BDI agents because "the fact that model builder and subject being modelled are referring to the same concepts does simplify matters".

Another approach when fine-grained data is not available is to rely on psychological theories of human behaviour to describe expected behaviour, and to use them to design the agent model. For instance, data about emotions is rarely available from field studies or statistics, so theories of emotions can help. In that case, BDI has the advantage of providing the right concepts for capturing these theories and formalise them in terms of mental attitudes. When there is neither data nor theory available, i.e. when the decision-making process is unknown, BDI is not an appropriate choice, as also noted by Wolfe et al. [START_REF] Wolfe | To bdi, or not to bdi: design choices in an agent-based traffic flow management simulation[END_REF].

To conclude, BDI agent models might at first seem harder to design because of the lack of fine-grained statistical data. But at the same time the BDI formalism allows modellers to exploit different types of data, such as observations from participatory modelling, or psychological theories, because of the straightforward matching between BDI concepts and the way in which people instinctively explain their behaviour.

Scalability

Another important criticism against the use of BDI agents in simulation is the computational weight of this approach. As it is already time-consuming to simulate only one BDI agent, with many reasoning components, it is intuitively hard to imagine that such a tool could be used for large-scale simulations with thousands of agents. Some works have integrated BDI agents in real-time environments such as games (see Section 3.3.3) and they tend to confirm this intuition. For example in [START_REF] Hindriks | Unreal goal bots. connecting agents to complex dynamic environments[END_REF], the authors develop bots using a BDI architecture in the game Unreal Tournament, and conclude that there are indeed problems of scalability and performance when the number of agents increases.

However, this technical drawback is bound to decrease and eventually disappear, as computational power of individual computers increases exponentially, and grid computing also allows the distribution of simulations on many computers to accelerate their execution even more. For instance Zhao et al. [START_REF] Zhao | BDI-based human decision-making model in automated manufacturing systems[END_REF] have built a BDI agent in a distributed environment, to partially replace the decision-making process of human beings in an automated system; this simulation only includes one agent so far but could technically be extended to many more agents. Nevertheless this appears to be a shift from one research issue to another, because the distribution of a simulator over a network is a very difficult problem too, that is far from being solved.

So scalability still limits the use of BDI agents in large-scale simulations for now, but the computational power of present personal computers already allows the use of BDI agents in smaller scale simulations. For larger scale simulations, when distribution is too complex or limited by a lack of hardware resources, the modeller can resort to an optimised platform, or optimise the implementation of their BDI engine. For instance, Cho et al. implemented a crowd simulator with BDI agents, and evaluated its performance in terms of computing time [START_REF] Cho | Bdi model-based crowd simulation[END_REF]. They showed that the processing time was roughly proportionate to the number of agents, and mostly dedicated to the belief update process applied every time a new belief is added. Their solution to minimise this computing time was therefore to optimise the reasoner by applying belief update rules only when their effect was significant for the current desire.

In addition, building simulations using BDI agents does not imply that all the agents should be cognitive. A wise choice of which agent should or should not be a BDI agent will have a huge impact on the the simulation scalability. Wolfe et al. investigate this "To BDI, or not to BDI" dilemma [START_REF] Wolfe | To bdi, or not to bdi: design choices in an agent-based traffic flow management simulation[END_REF] in the field of aerial traffic management simulations. They mention that the "BDI paradigm is an excellent choice when modeling well-understood decision making", but notice efficiency issues when using BDI agents in large-scale simulations. They suggest that one solution is to choose wisely which agents to model as BDI agents (pilots in their example), and which agents could have simpler architectures (planes, etc). This mixed population of BDI and not-BDI agents increases the performances of their simulator. Novak et al. implemented BDI reasoning capabilities in autonomous military robots teams for surveillance missions [START_REF] Novák | Simulated multi-robot tactical missions in urban warfare[END_REF], to improve flexibility compared to a simple reactive planner. However they noted that these agents were heavier and should thus be limited to scenarios where complex deliberation was actually needed.

To conclude, BDI architectures are indeed heavier due to the more complex reasoning processes of the agents, so they have to be used wisely. Many authors have successfully applied them in their simulations, by optimising the computation inside each agent, and by limiting their use to agents and scenarios where they were actually justified, mixing BDI agents with simpler agents.

Limited (conceptual) expressivity

As highlighted in Section 2.3.4, the BDI architecture has the advantage of being easily extensible with more concepts than the three basic ones (beliefs, desires, intentions). Nevertheless, before being integrated in a BDI agent, the properties of any additional concept and its links with the existing concepts (at least beliefs, desires and intentions, but possibly others in an already extended BDI architecture) should have been deeply studied.

DSTO (Australian Defence) programmers have listed a number of features that need to be added to the BDI framework so that developers do not have to explicitly implement them in every model, and so that applications can be less constrained. Norling quotes for example emotion, fatigue, learning or memory [102, p.3]. We could also extend this list with group attitudes and social commitments. In particular, in a society of agents, the BDI architecture can efficiently be used to represent an individual reasoning process, but only few formalisms have been developed for the reasoning process of social groups. Actually there is no issue metaphorically using a BDI architecture for an agent representing a group of agents when it is well-defined offline (before the beginning of the simulation). But there are only a few studies on how to attribute mental states to social groups of agents emerging online, during the simulation, when the attitudes of the group should interact with the attitudes of the agents composing it.

Only few theoretical studies have been led about the above concepts. There is thus a huge conceptual lack here. Nevertheless we stay optimistic and assume that in the next years many researchers will try to tackle these issues. For example, recent works have begun to analyse the links between BDI and emotions [START_REF] Adam | The emotions: from psychological theories to logical formalisation and implementation in a BDI agent[END_REF] or social attitudes [START_REF] Gaudou | Formalizing social attitudes in modal logic[END_REF]. Such work has to be continued to provide a clear, well-grounded and standardised extended BDI framework.

Learning

One of the major a priori criticisms of the BDI architecture of agents is its inadequacy to support learning mechanisms: "one weakness that the BDI agent model shares with other human operator models (agent-based or otherwise) is its inability to support agent learning" [159, p.9]. This is particularly due to the fact that most powerful learning algorithms are dedicated to quantitative learning (learning numerical values, e.g. in neural network...) whereas symbolic learning is a much harder issue.

Nevertheless there exist some extensions of the BDI architecture aiming at integrating a learning capability into BDI agents. For example, in the video game Black and White [START_REF] Molyneux | Postmortem: Lionhead studios' black and white[END_REF], the main character has a BDI architecture extended with a reinforcement learning capability to allow the player to teach their avatar. In another example, [102, p.4] extended BDI with the RPD (Recognition-Primed Decision) strategy used by experts of a domain: the agent progressively learns (from its mistakes) to make very fine distinctions between situations, to allow a nearly automatic plan selection based on it. These examples show that in contrast to preconceived ideas, BDI agents can be useful in simulations even when agents need learning capabilities.

Criticisms

Conclusion

To conclude this section, we have shown that BDI agents can bring important benefits to social simulation in some cases, depending on the goal of the simulation, the required features of the agents, the desired observation scale, and the field of application. We have also discussed why BDI agents are not used more often in social simulations, and shown that there are solutions to overcome these obstacles. One more obstacle that we have not discussed yet follows from the high level of abstraction provided by BDI models: this paradigm is a special way of thinking about an autonomous agent, not in terms of classes and methods, but in terms of mental attitudes. As a result, it can be hard for designers to conceptualise their agent model, all the more when they are not programming experts but scientists from other disciplines (epidemiologists, geographers, sociologists, etc), as it is often the case in this very multidisciplinary field. To facilitate the use of BDI agents in social simulations, it is therefore essential to provide designers with methodologies and tools to support the modelling process and the implementation of the simulator. This is discussed in the next section.

We have seen above that BDI agents bring many benefits to social simulations but that some obstacles impede their wider use. This section tries to provide a guide for concretely integrating BDI agents into social simulations, and to overcome these obstacles.

Examples of existing simulations with BDI agents

In this article we have presented a huge array of existing agent-based simulations that use BDI agents. We can first notice that the number of platforms and tools used to implement and run these models is very high. For example, DamasRescue [START_REF] Paquet | DAMAS-Rescue Description Paper[END_REF] models rescue workers after a natural crisis, implements them with Jack agents [START_REF] Busetta | Jack intelligent agents-components for intelligent agents in java[END_REF], and develops an ad hoc simulator to run the simulations. Lui et al. [START_REF] Lui | An architecture to support autonomous command agents for onesaf testbed simulations[END_REF] use Jack Teams, an extension of the Jack framework for teams of agents, to model military commanders in land operations scenarios. [START_REF] Bosse | Towards integration of biological, psychological and social aspects in agent-based simulation of violent offenders[END_REF][START_REF] Bosse | Integrating rational choice and subjective biological and psychological factors in criminal behaviour models[END_REF] the influence of psychological (emotions) as well as social and biological aspects on criminal behaviour; they use their own language, called LEADSTO, that allows to model both qualitative BDI aspects and quantitative biological aspects.

These few examples are enough to show that many simulations do use BDI agents, but also that most authors have needed to develop at least part of the simulator, be it the BDI engine or the simulation engine. This approach is time consuming and inefficient, involving the re-development from scratch of ad hoc tools instead of reusing existing ones. This could be explained by the lack of awareness of existing tools in other communities, or the greater cost of using them compared to creating ad hoc tools.

BDI platforms that support simulation

BDI platforms provide many useful features (intention selection mechanisms, etc) that we do not want to have to reprogram in each agent model. However, they miss other necessary features for simulation (e.g. visualisation). For example, Jack Intelligent Agent are the basis of JackSim [START_REF] Ronald | Evaluating jack sim for agent-based modelling of pedestrians[END_REF] and its extension to teams [START_REF] Lui | An architecture to support autonomous command agents for onesaf testbed simulations[END_REF], which have been used to develop simulators. But all the simulation part was implemented from scratch, or at best coupled with an external visualisation tool.

This example shows that developing a simulator from a BDI platform is not an easy task and requires a lot of work, in particular related to the implementation and management of the environment, the visualisation of the simulation, and the time and scheduling of agents. The management of data is also required for most modern simulators: the simulator should be able to import tabular or geographical data, to create and initialise agents from this data, and to store data in files. Most of these features can now be found in agent-based modelling and simulations platforms such as GAMA [START_REF] Grignard | Gama 1.6: Advancing the art of complex agent-based modeling and simulation[END_REF], Netlogo [163], Repast Simphony [START_REF] North | Complex adaptive systems modeling with repast simphony[END_REF] or Mason [START_REF] Luke | Mason: A multiagent simulation environment[END_REF], but are not natively provided by BDI frameworks.

Simulation platforms that support BDI

In agent-based simulation platforms cited above such as Netlogo, the architecture describing the behaviour of agents is often very simple. Uhrmacher et al. "shows how trivial the agent behavior is in the type of social simulation that some researchers have had to content themselves with, due to the limitation of agent-based simulation toolkits" [START_REF] Uhrmacher | Multi-Agent Systems: Simulation and Applications, chapter Agent-Based Simulation Using BDI Programming in Jason[END_REF]. But agent-based simulation toolkits are now evolving, and in particular they sometimes provide more complex architectures, such as the BDI architecture.

For example, Sakellariou et al. have extended the NetLogo platform [163] by providing libraries that allow the easy development of BDI agents exchanging FIPA-ACL messages in a FIPA protocol [START_REF] Sakellariou | Enhancing netlogo to simulate bdi communicating agents[END_REF]. They developed a demonstration example of a forest fire simulation where firemen agents cooperate and coordinate under the Contract Net Protocol.

Recently, the GAMA platform has also been extended to allow modellers to use a BDI architecture for their agents [START_REF] Caillou | A simple-to-use bdi architecture for agent-based modeling and simulation[END_REF]. Without being as sophisticated as a dedicated BDI platform, it still provides primitives to manage beliefs, desires and intentions databases in agents and to describe their behaviour in terms of the plans they can perform. The focus of this extension was to keep the implementation simple for modellers, even non-computer scientists.

These examples illustrate a trend to extend simulation platforms with an ad hoc BDI module. However, it is not optimal to reprogram BDI reasoning from scratch, while dedicated BDI platforms have been developed and improved over many years. It would therefore be more efficient to connect simulation platforms with existing BDI platforms. For example, Caballero et al. have proposed an integration of JASON into the MASON platform [START_REF] Caballero | Using cognitive agents in social simulations[END_REF].

Nevertheless in order to avoid that each simulation platform develops its own connectors to each BDI platform, Padgham et al. have proposed a framework aiming at combining any simulation platform with any BDI platform [START_REF] Padgham | Integrating bdi reasoning into agent based modeling and simulation[END_REF].

Their framework already allows to combine several BDI platforms (e.g. Jason and Jack) with several simulation platforms (e.g. Matsim 3 and Repast). It can be used in a simulation platform to allow agents to delegate their reasoning to a BDI framework. But it can also be used as the master component, driving the simulation platform and synchronising its execution with the BDI platform. It is a very promising tool that could speed up the use of BDI agents in agent-based simulations.

Table 4 considers all the platforms that have been compared in [START_REF] Kravari | A survey of agent platforms[END_REF]

Methodologies to help with creating BDI agents

In Agent Oriented Software Engineering (AOSE), dedicated methodologies and tools exist for expert programmers, such as the Prometheus methodology and associated tool with Jack code generation [START_REF] Padgham | Prometheus: A methodology for developing intelligent agents[END_REF][START_REF] Padgham | Prometheus design tool[END_REF]; [START_REF] Cossentino | A notation for modeling jason-like bdi agents[END_REF] describe an UML like notation and Jason-code generation. These methodologies are sometimes used in simulations, for instance [START_REF] Sokolova | An agent-based decision support system for ecological-medical situation analysis[END_REF] use Prometheus methodology for modeling agents with a BDI architecture, in a decision-support system for evaluating the impact of pollutant exposure on population health. But while methodologies do exist for BDI agents in AOSE, they are aimed at computer scientists and expert programmers. On the contrary, social simulations are often developed by non-computer scientists, who need appropriate methodologies and tools to help them. In the social simulation field, very few methodologies exist and are used, and existing ones are too high level. For example the ARDI method [START_REF] Etienne | Ardi: a co-construction method for participatory modeling in natural resources management[END_REF] focuses on the elicitation from stakeholders of a description of the reference system in terms of actors, resources and dynamics; it does not go any deeper in the design of agents and their behaviours. Graphical visualisation tools could also help in involving stakeholders and the general public in participatory modeling [START_REF] Gaudou | Toward a methodology of collaborative modeling and simulation of complex systems[END_REF].

Due to the lack of dedicated methodologies, UML is most often used to conceptualise the agent model. Recently, the Tactics Development Framework [START_REF] Evertsz | Tactics development framework (demo)[END_REF][START_REF] Evertsz | Agent oriented modelling of tactical decision making[END_REF] was introduced as a methodology for eliciting tactical knowledge from military experts and embedding it in agent-based simulations, and was shown to be better than UML. It was then adapted to modeling the behaviour of the civilian population in Australian bushfires [START_REF] Adam | SWIFT: simulations with intelligence for fire training[END_REF].

To conclude, technologies start to appear to link BDI platforms with modelling and simulation platforms. Their aim is to facilitate the design and implementation by any modellers (especially non computer scientists) of simulations embedding complex (in particular BDI) agents. Nevertheless there is still a lack of methodologies to drive the modelling process.

Conclusion

We have so far shown the benefits associated with using the BDI architecture for modelling agent behaviour in ABMS. We have talked about the capabilities provided by the core architecture: adaptability, robustness, abstract programming and the ability to explain their behaviour. We have noted that BDI is rarely used in implemented ABMS and we have examined some of the possible reasons for these in turn: the field of application -BDI is unsuitable for modelling simple entities such as bacteria, but ideal for modelling humans; the purpose of the simulation -as recognised by Axelrod, when the simulation requires an accurate representation, complexity is inevitable; the scalability limitations of BDI -these were refuted by real-time game applications; or conceptual lacks of BDI -these can be corrected through extensions such as adding emotions or norms.

Despite the clear benefits, and the lack of good reason against using BDI (at least in some circumstances) there is a lack of uptake of the paradigm in the ABM community. We suggest that this may be because of the lack of appropriate tools to help designers integrate BDI agents into their simulations, and more specifically, the lack of BDI modelling capabilities in existing simulation and modelling platforms. As it is put by [START_REF] Uhrmacher | Multi-Agent Systems: Simulation and Applications, chapter Agent-Based Simulation Using BDI Programming in Jason[END_REF]: "the example shows how trivial the agent behavior is in the type of social simulation that some researchers have had to content themselves with, due to the limitation of agent-based simulation toolkits". We aim to address this problem in future work by extending the GAMA platform to allow the creation of BDI-style agents.

Table 1 :

 1 Summary of benefits and drawbacks of BDI architecture w.r.t. agents characteristics (learning will be discussed in Section 3.5.4)

		Importance
	characteristics	of BDI	Benefits and drawbacks
	Representation	+++	Non-ambiguous representation
	and reasoning		Theory Of Mind (reasoning about others)
			Limited basic expressivity overcome by extensions
			Complex reasoning: small-scale simulation
	Self-explanation	+++	Intuitive explanation (folk psychology)
	Communication	++	High-level communication (coordination)
			Mentalistic semantics of ACL
			Flexible interaction protocols
	Planning	++	Stable, robust decision procedure
			Complex (heavy) but flexible deliberation
	Emotions	+	Formal description, links with mental attitudes
			Reasoning and interpreting (reverse appraisal)
			Influence on behaviour (coping strategies)
	Norms	+	Explicit representation allows reasoning
			Integration in decisions-making
			Handling of pre-established norms only
	Decision-making	-	Heterogeneous decisions, inter-agent differences
			Irrational decisions, influence of psychological factors
			Practical reasoning
	Learning	-	Symbolic learning, not standard in BDI

Table 2 :

 2 Summary of interests of BDI w.r.t. agents characteristics

		Interest
	purpose	of BDI	Benefits	Drawbacks
	Prediction	Medium	Realistic, adaptable micro-level behaviour	Complexity, scalability
			possibly irrational individual cognition	Requires detailed data
	Performance	High	Right level of abstraction of human-like behaviour
	of a task		Awareness, cooperation in mixed human-agent teams	Harder design,
			Modular, scalable, flexible design	uncommon paradigm
	Training	High	Accurate realistic behaviour for better immersion
			Adaptability to a dynamic environment	Harder design,
			Descriptivity	uncommon paradigm
	Entertainment Medium	Rich plausible human-like behaviour: immersion, challenge	Scalability, performance
			Quick decision under uncertain and incomplete information	Harder design vs scripts
			Right abstraction level to capture real players' strategies
	Education	Medium	Intuitive explanations of behaviour	Unneeded realism & complexity
			via built-in folk psychology concepts (B,D,I)	for non essential agents
	Proof	Low	Realistic cognition required to prove micro-macro	Unneeded realism & complexity
			links and complex socio-cognitive phenomena	to prove simpler hypothesis
	Discovery	Low	Realistic fine-grained behaviour model to discover	Harder understanding and deduction
			non-intuitive effects and micro-macro links in	Harder specification of decision rules
			adaptive dynamic complex systems

Table 3 :

 3 Summary of criticisms against BDI and proposed solutions .

	Solutions

  Cho et al. use an existing 3D game engine (SourceEngine) but developed an ad hoc BDI reasoner to implement NPC extinguishing a forest fire in a virtual world. Mcilroy et al. model fighter pilots by using the distributed Multi-Agent Reasoning System (dMARS) [45], a realisation of the Procedural Reasoning System, an agent architecture based on the BDI framework. Bosse et al. investigate in

Table 4 :

 4 and detail how they support BDI agents. Support of Agent-Based platform (list of platforms from[START_REF] Kravari | A survey of agent platforms[END_REF]).

	BDI agents support	ABM platforms
	Full native	Agent Factory, AgentBuilder
	support	INGENIAS Development Kit, Jack, Jadex, Jason
	Full support	AgentScape (Jason), Jade (Jadex, Jason),
	with coupling	EMERALD (based on Jade), JAMES II
		MadKit, SeSAm (through Jade)
	Limited	Netlogo (extension), GAMA (plugin)
	support	Repast Simphony, Mason
	No support,	A-Globe, AnyLogic, Cormas, Cougaar, CybelePro
	No available information	JAS, Swarm
	Not relevant	JIAC (support planned for Spring 2016)

The expression "cognitive architecture" used here refers to architectures based on cognitive theories, such as[START_REF] Sun | Duality of the Mind[END_REF][START_REF] Newell | Unified theories of cognition[END_REF][START_REF] Anderson | The architecture of cognition[END_REF]. It should not be confused with the term "cognitive agents" that is usually synonym to "intelligent agents" in the MAS literature (in this wide sense, agents with BDI architecture are also considered cognitive).

http://www.nyu.edu/ccpr/laser/plancinfo.html

Acknowledgements

This work has been partially funded by the ACTEUR ("Spatial Cognitive Agents for Urban Dynamics and Risk Studies") research project. The ACTEUR project is funded by the French National Research Agency (ANR) under grant number ANR-14-CE22-0002.