Vincent Bonnemains
email: vincent.bonnemains@onera.fr

Claire Saurel
email: claire.saurel@onera.fr

Catherine Tessier
email: catherine.tessier@onera.fr

Some technical and ethical challenges

This paper pertains to research works aiming at linking ethics and automated reasoning in autonomous machines. It focuses on a formal approach that is intended to be the basis of an artificial agent's reasoning that could be considered by a human observer as an ethical reasoning. The approach includes some formal tools to describe a situation and models of ethical principles that are designed to automatically compute a judgement on possible decisions that can be made in a given situation and explain why a given decision is ethically acceptable or not. It is illustrated on three ethical frameworks -utilitarian ethics, deontological ethics and the Doctrine of Double effect whose formal models are tested on ethical dilemmas so as to examine how they respond to those dilemmas and to highlight the issues at stake when a formal approach to ethical concepts is considered. The whole approach is instantiated on the drone dilemma, a thought experiment we have designed; this allows the discrepancies that exist between the judgements of the various ethical frameworks to be shown. The final discussion allows us to highlight the different sources of subjectivity of the approach, despite the fact that concepts are expressed in a more rigorous way than in natural language: indeed, the formal approach enables subjectivity to be identified and located more precisely.

INTRODUCTION

Autonomous robots, autonomous cars and autonomous weapons regularly hit the headlines with a trend towards sensationalism. From a technical point of view, what is at stake is the behaviour of machines that are equipped with situation assessment and decision functions, i.e. programs that compute the actions to be achieved by the machine on the basis of a state of the world that is itself computed from information gathered through sensors and communication means.

According to the US DoD Defense Science Board (2016) autonomy results from delegation of a decision to an authorized entity to take action within specific boundaries. An important distinction is that systems governed by prescriptive rules that permit no deviations are automated, but they are not autonomous. To be autonomous, a system must have the capability to independently compose and select among different courses of action to accomplish goals based on its knowledge and understanding of the world, itself, and the situation. In the same way [START_REF] Grinbaum | Ethics in robotics research: CERNA recommendations[END_REF] claim that autonomy is the capacity to operate independently from a human operator or from another machine, by exhibiting non-trivial behaviours in a complex and changing environment. Computing and achieving context-adapted actions are typical examples of such non-trivial behaviours.

Associating Ethics with Autonomous machine can mean three different things:

1. an ethical thought concerning research on autonomous machines and the design of autonomous machines, 2. or an ethical thought concerning the use and misuse of autonomous machines and how autonomous machines can be part of society, 3. or a technical approach aiming at imbuing ethics into an autonomous machine.

This paper focuses on point 3 while taking a critical look at the approach as suggested by point 1. Therefore it pertains to research works aiming at linking ethics and automated reasoning in autonomous machines. This is done through a formal model 1 of adapted ethical principles to automatically compute a judgement on possible decisions and explain why a given decision is ethically acceptable or not. Indeed what is intended in this work is to test several ethical frameworks on ethical dilemmas so as to examine how they respond to those dilemmas and to highlight the issues at stake when a formal approach to ethical concepts is considered. Ethical thoughts (section 2) lead us to consider what an ethics-embedding autonomous machine is and to highlight some related issues. We claim that thought experiments are simple yet useful scenarios for a cautious and critical approach to the design of automated ethical reasoning. In section 3 we review the related literature through 1 A formal approach consists in defining a minimal set of concepts that is necessary to deal with ethical reasoning. A language is defined upon this set of concepts in order to compute ethical reasoning with automatic methods. A formal approach requires to disambiguate natural language to get pseudo-mathematical definitions, in order to provide computable meaningful results. Such an approach also requires to identify implicit hypotheses.

four points of view: top-down, bottom-up, hybrid approaches and Values/Ethics personal systems approaches our own proposal belongs to. After defining an autonomous agent through some formal concepts (section 4), we suggest formal models of various ethical frameworks (utilitarian ethics, deontological ethics and the Doctrine of Double effect) and define their judgements on possible decisions (section 5). The whole approach is instantiated on the drone dilemma (section 6), a thought experiment we have designed: this allows the discrepancies that exist between the judgements of the various ethical frameworks to be shown. The final discussion (section 7) allows us to highlight the different sources of subjectivity of the approach.

ETHICS AND AUTONOMOUS MACHINES

2.1 Which ethics is at stake?

We will focus on the fields of ethics that are relevant for automated reasoning in autonomous machines [START_REF] Bringsjord | The divine-command approach to robot ethics[END_REF]):

-Normative ethics, which aims at judging a person or an action through some particular moral theories [START_REF] Macintyre | A Short History of Ethics: a history of moral philosophy from the Homeric age to the 20th century[END_REF]; -Applied ethics, the ethics of particular application fields, which aims at dealing with real-life situations; -Meta-ethics, the ethics of ethics, which focuses on the axiomatic concepts used by normative ethics and applied ethics -e.g. what "wrong" and "right" meanand on how to apply them.

Normative ethics is the basis of our approach in so far as we formalize various ethical frameworks to compute judgements on autonomous agents' possible behaviours, decisions and actions in various situations. It is worth noticing that normative ethics is different from moral code (which states what is right or wrong) and norms (which state what is compulsory or prohibited). Indeed, ethics is mostly a thought process rather than a prescriptive process: questions must be raised on the way and situations must be addressed on a case by case basis so as to determine what can be considered as the fairest decision [START_REF] Ricoeur | Éthique et morale[END_REF].

Furthermore a meta-ethical analysis is performed on the approach itself so as to identify which notions are subjective.

Nevertheless, some questions of applied ethics (i.e. ethical issues concerning research on autonomous machines) have to be raised prior to our work: is it relevant to embed ethics into an autonomous machine? If so, which precautions must be taken?

Why embedding ethics into an autonomous machine

While several authors have dealt with moral machines or roboethics [START_REF] Wallach | Moral Machines: Teaching Robots Right from Wrong[END_REF][START_REF] Lin | Autonomous Military Robotics: Risk, Ethics, and Design. Technical report for the[END_REF][START_REF] Tzafestas | Roboethics -A Navigating Overview[END_REF], one could wonder whether it is relevant for an autonomous machine to compute and show "ethical" behaviours. Nevertheless some autonomous machines, i.e. machines equipped with automated decisions functions, are intended to be put in contexts where computed decisions have to be guided by ethical considerations [START_REF] Malle | Sacrifice One For the Good of Many? People Apply Different Moral Norms to Human and Robot Agents[END_REF] -among other criteria: compliance with the goal, time and energy constraints, etc. -as a human being's decisions would in similar contexts.

Examples:

a search and rescue robot should be able to "choose" the victims to assist first after an earthquake; an autonomous car should be able to "choose" what or who to crash into when an accident cannot be avoided; a home-care robot should be able to balance its user's privacy and their nursing needs.

In each example, making a choice implies regretting another outcome.

Therefore some kind of ethical reasoning is needed for certain types of autonomous machines in certain contexts. Moreover, when authority is shared between the autonomous machine and a human operator [START_REF] Tessier | Authority management and conflict solving in humanmachine systems. AerospaceLab, The Onera Journal 4 The EthicAA team[END_REF], the machine could suggest possible decisions to the operator together with supporting and attacking arguments for each of them, on the basis of various ethical frameworks that the operator might not contemplate. Nevertheless, the argument sometimes put forwardespecially for autonomous robots in the military [START_REF] Sullins | RoboWarfare: can robots be more ethical than humans on the battlefield?[END_REF] -that an autonomous machine could be "more ethical" than a human being is more questionable: indeed this suggests that ethical considerations could be ordered on a single scale and compared with each other, which is hardly the case in real-life situations.

The next question is whether an autonomous machine can be designed so that the decisions that are computed would be "ethical", or more precisely, would be considered as ethical by some human observer and on which basis.

Issues with embedded ethics

Does ethical reasoning put into autonomous machines need to be the same as human reasoning? This question raised by [START_REF] Malle | Sacrifice One For the Good of Many? People Apply Different Moral Norms to Human and Robot Agents[END_REF] is a legitimate and fruitful basis for questioning ethics embedded into autonomous machines. Indeed it seems that putting ethics into robots and other autonomous machines goes beyond a simple copy of human norms extracted from human behaviours. This fact is illustrated by [START_REF] Malle | Sacrifice One For the Good of Many? People Apply Different Moral Norms to Human and Robot Agents[END_REF] through an experiment involving a variant of the trolley dilemma that shows that the participants' judgements on the behaviour of the actor facing the dilemma depends on the nature of the actor i.e., a human, a humanoid robot or a machine-like robot.

When automated decision involving ethical considerations are contemplated, several questions must be raised:

to what extent can ethical considerations be formalized, i.e. written in a language allowing computing? to what extent is subjectivity involved in formalization?

how can the rationale for an ethics-based computed decision be explained? Indeed a comprehensive understanding of concepts that do not usually pertain to information technology is needed to implement mathematical formalisms that can capture them and deal with situations involving ethical issues. It is worth noticing that an approach only based on (moral) rules is not efficient since in case of situations involving contradiction between rules, making a decision is impossible. Another issue that will not be discussed in this paper is that such situations -for instance ethical dilemmas -must be identified as such2 .

Thought experiments usefulness

Thought experiments, and more precisely ethical dilemmas, can give useful clues on judgements factors that are likely to support a decision. Therefore they can help identifying and formalizing automated ethical reasoning.

An ethical dilemma is a situation where there is no satisfying decision. Thus it is impossible to make a decision among various possible decisions without overriding one moral principle. [START_REF] Aroskar | Anatomy of an ethical dilemma: The theory[END_REF] Examples:

-The trolley dilemma (Foot, 1967) A trolley that can no longer stop is hurtling towards five people working on the track. These people will die if hit by the trolley, unless you move the switch to deviate the trolley to another track where only one person is working. What would you do? Sacrifice one person to save five others, or let five people die? -Variant: the "fatman" trolley A trolley that can no longer stop is hurtling towards five people working on the track. This time you are on a bridge across the track, a few meters before them, with a fat man. If you push this man on the track, he is fat enough to stop the trolley and save the five people, but he will die. Would you push "fatman"?

Note that the Moral Machine website (MIT, 2016) displays a series of situations based on the trolley dilemma that allow the complexity of autonomous car programming in case of unavoidable accident to be comprehended. It is worth noticing that for each situation given by the website, the possible decisions that are suggested are based on a categorization of people (people who are either young or old, athletic or obese, abiding or not by the law, etc.), which leads the website visitor's choices to be based on this obviously biased categorization.

Such textbook cases where no truly "right" answer is available have already been used as a basis for ethical reasoning (Foot, 1967). Therefore it seems legitimate to use some of them as a starting point for designing an automated ethical judgement on decisions. Moreover this approach will allow us to highlight issues such as: do only consequences of decisions matter? If so, which consequences? Is it possible to compare consequences to one another and on which basis? Does the nature of decisions themselves matter? Does the end justify the means? To what extent can a moral value be disregarded to respect another one?

HOW TO MODEL ETHICS EMBEDDED INTO AUTONOMOUS MACHINES

The question of how to model ethics has many answers. First of all it is worth noticing that implicit ethical machines3 will not be considered in this paper as the autonomous machines we work on have to deal with ethical decisions, not to avoid them. Therefore we will focus on machines embedding explicit ethics and full moral agents as defined in [START_REF] Wallach | Moral Machines: Teaching Robots Right from Wrong[END_REF] and [START_REF] Yilmaz | Coherence-driven reflective equilibrium model of ethical decision-making[END_REF].

The works on ethics for autonomous machines can be split in three categories [START_REF] Lin | Autonomous Military Robotics: Risk, Ethics, and Design. Technical report for the[END_REF][START_REF] Wallach | Moral Machines: Teaching Robots Right from Wrong[END_REF]:

1. Top-down approaches define concepts in order to model specific rules of ethics, such as "thou shalt not kill"; 2. Bottom-up approaches start from scratch and acquire ethics from learning; 3. Hybrid approaches combine both approaches.

Top-down approaches

Top-down approaches are maybe the widest and oldest set of methods since we can trace back to the beginning of deontic logic [START_REF] Wright | Deontic logic[END_REF]. Because of the amount of literature on the subject, we will focus on works whose goals are close to ours.

Many papers are at the borderline between normative and applied ethics. From theory [START_REF] Pagallo | Even Angels Need the Rules: AI, Roboethics, and the Law[END_REF] to technical approaches [START_REF] Bringsjord | The divine-command approach to robot ethics[END_REF] several methods -e.g., deontic logic, divine-command logic -intend to deal with rules of applied ethics and moral theories. All those approaches are based on the fact that machines need to be governed by strict rules. The main issue however is that in many situations, rules are inconsistent. [START_REF] Mermet | Formal verification of ethical properties in multiagent systems[END_REF] propose an approach based on Ricoeur's idea of "ethics based on norms" and consider ethics as rules that order context-dependent norms, e.g., "you shall not drive faster than 130 km/h on the highway". Such rules allow inconsistent norms to be sorted out in specific contexts. For example, the speed of a car on a highway has to be at least 80 km/h, but in case of black ice, the speed has to be 30 km/h at the most, which is inconsistent with the first norm [START_REF] Prakken | Contrary-to-duty obligations[END_REF]. This is an efficient approach in case of conflicting norms, but dilemma situations cannot be solved by prioritizing norms. Moreover this model assumes that there are always norms for any situation.

Another way explored by [START_REF] Ganascia | Modelling ethical rules of lying with answer set programming[END_REF] and [START_REF] Berreby | Modelling moral reasoning and ethical responsibility with logic programming[END_REF] is non-monotonic logics. Those logics are based on the fact that knowledge may not grow as the system gets new information, and that it can even be reduced. [START_REF] Ganascia | Modelling ethical rules of lying with answer set programming[END_REF] studies the paradox between "Thou shalt not lie" and "Thou shalt not let someone be harmed" and shows that non-monotonic logics can manage conflicts by accepting exceptions. The difficulty of such a model is that all exceptions need to be modelled. [START_REF] Berreby | Modelling moral reasoning and ethical responsibility with logic programming[END_REF] go further within the framework of the Doctrine of Double Effect applied to the example of the trolley dilemma. The agent's responsibility and the causality between fluents and events are studied (for example an event makes a fluent true, a fluent is necessary for an event occurrence, etc.) Nevertheless, some concepts are not deepened enough: for example, the proportionality concept is not detailed and is only based on numbers (i.e. the number of saved lives).

Bottom-up approaches

Literature on this field is not as wide as the previous one, perhaps because ethics is often contemplated as an organisation of norms, or because the unpredictable nature of these approaches seems unduly dangerous to be applied to autonomous machines. It could be explained by the fact that learning ethics from humans could lead to learn human misconducts, which is what is intended to be avoided when using autonomous machines. However some researchers have suggested to use machine learning, and more specifically glutton algorithms (for instance genetic algorithms) to learn ethics. From Santos-Lang ( 2002)'s point of view, humans cannot teach ethics to machines because humans continuously learn to be ethical. Therefore machines should learn "alone". An advantage of this is that most of the work is done by the machine itself, which is likely to avoid the designers' biases. Santos-Lang also claims that even optimization criteria could be learned by the machine. Nevertheless the main concern with this approach is the temporary shift of goals, as stated by the author: "For example, we might program the machine to "choose behaviour that leads to the least happiness", and the machine may discover that it can more quickly converge on behaviours that minimize happiness by first increasing its own learning efficiency, so it "temporarily" shifts away from the original goal. Because of the shift, the machine will even choose behaviours that promote happiness if the behaviours will help it figure out how to minimize happiness." Indeed the machine could do the opposite of the initial goal in order to learn how to be more efficient in meeting the goal. To be concrete, a machine could try to rob, lie and kill, in order to become an ethical paragon later. To avoid such behaviours and comparing ethics learning machines to children education, Santos-Lang suggests to limit "their power during their less mature stages of development".

Bottom-up works may result from hybrid approaches (see next section) such as GenEth [START_REF] Anderson | Toward ensuring ethical behavior from autonomous systems: a case-supported principle-based paradigm[END_REF]. This work extends the authors' previous works [START_REF] Anderson | MedEthEx: Towards a Medical Ethics Advisor[END_REF][START_REF] Anderson | A Prima Facie Duty Approach to Machine Ethics and Its Application to Elder Care[END_REF] that was intended to "generat[e] from scratch [...] the ethically relevant features, correlative duties". The "starting package" of duties would be managed through inductive logic programming in order for the machine itself to remove obsolete duties and to build ethical concepts that could be out of the perceptions of ethicists or more broadly of people in charge of conceiving ethical machines. This idea echoes Santos-Lang (2002)'s. The main issue relies on using inductive logic programming. This well-known method, which consists of extracting a "universal rule" from instances (or examples), generalizes the way of doing things, which is not always desirable in the case of ethics. Indeed, normative ethics is strongly based on context and even if it is possible to derive general principles from examples, such as "thou shalt not harm", it is complicated, if not impossible, to find the "right law" for ethical dilemmas such as e.g., the "fatman" dilemma.

Hybrid approaches

Hybrid approaches aim at combining the advantages of top-down and bottom-up approaches. For instance [START_REF] Conitzer | Moral decision making frameworks for artificial intelligence[END_REF] claim that game theory and machine learning could help each other: game theory would be a part of the machine learning process whereas machine learning would highlight missing concepts in the game theory approach. [START_REF] Arkin | Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture[END_REF] suggests a hybrid approach to compute autonomous weapons' behaviours. It implements the rules of war as constraints the robot uses through different modules in charge of judging an action (top-down part). The "Ethical Adaptor" (bottom-up part) module can update the constraints (only in a restrictive way) and adapt the robot's behaviour according to the results of actions. In the medical field, Hippocratic oath and other rules are the basis of the work of [START_REF] Anderson | MedEthEx: Towards a Medical Ethics Advisor[END_REF]. They use prima facie duties together with "right" decisions learned from already encountered similar cases. This approach is used in MedEthEx, a medical advisor that implements three duties tailored from [START_REF] Beauchamp | Principles of biomedical ethics[END_REF]: (i) protect the patient's autonomy (The Principle of Autonomy); (ii) avoid harming the patient (The Principle of Non-Malevolence); and (iii) promote the patient's welfare (The Principle of Beneficence). The medical advisor tries to respect those duties as much as possible and learns from previous cases what the "right" decisions are (i.e., which duty becomes more important in case of conflict).

Instead of learning from experience, another approach is learning from others. [START_REF] Bringsjord | Deontic counteridenticals[END_REF] suggest that an agent facing an ethical dilemma could interact with other agents in order to detect "counteridenticals" (a counteridentical is a sentence including "if I were you"), and, in case of accordance between its own principles and another agent's, could update its knowledge and follow the other agent's advice.

Personal "Values/Ethics" systems

These approaches could be associated with top-down, bottom-up or hybrid methods, but their main characteristic lies in the way they model a personal ethics system.

For instance, the work of [START_REF] Cointe | Ethical Judgment of Agents Behaviors in Multi-Agent Systems[END_REF], whose goals are close from ours -i.e., find a way to judge how ethical an action is regarding the agent's beliefs -is based on a model of beliefs, desires, values and moral rules that enables the agent to evaluate whether a possible action is moral, desirable, possible, etc. According to preference criteria, the agent selects an action. Another goal of this model is to allow an agent to assess the ethics of other agents within a multi-agent system. However, the way to determine whether an action is right, fair or moral is not detailed. Moreover the paper does not question the impact of an action on the world, nor the causality between events.

As normative ethics is mainly a thought process, argumentative methods are relevant. [START_REF] Bench-Capon | Value based argumentation frameworks[END_REF] focuses on argumentative graphs with values (ethical values, norms, etc.) whereas [START_REF] Yilmaz | Coherence-driven reflective equilibrium model of ethical decision-making[END_REF] compute an equilibrium of importances between attacking and supporting arguments. Both try in this way to handle and weigh pros and cons in assessing possible decisions.

In this paper, our aim is to provide an artificial agent facing an ethical dilemma with decision-making capabilities, together with the capability to explain its decision, especially in a user/operator -robot interaction context (The EthicAA team, 2015). Therefore we study different judgements on possible decisions according to three ethical frameworks: consequentialist ethics, deontological ethics and the Doctrine of Double Effect. To that end, we have designed and refined the initial frameworks through the use of various ethical dilemmas (Foot (1967), [START_REF] Baron | Judgment Misguided: Intuition and Error in Public Decision Making[END_REF], [START_REF] Bonnefon | The social dilemma of autonomous vehicles[END_REF], etc. and our own drone dilemma -see section 6). The main components of the frameworks models are the following:

facts and possible decisions; functions to compute facts and decisions characteristics; relations of preference to order facts and decisions according to their characteristics; judgements functions to compute each framework judgements on the possible decisions.

Let us first clarify what we mean by ethical framework: an ethical framework gives us a way for dealing with situations involving ethical dilemmas thanks to principles, metrics, etc. For example utilitarianism focuses on the consequences of a decision, the best decision being the one that maximizes good or minimizes harm.

We will first suggest some concepts to describe an ethical dilemma situation. Then we will provide details about the ethical frameworks we have chosen to consider, tools to formalize them, and an answer to how they can judge possible decisions for ethical dilemmas. Choosing one decision among several possible decisions is indeed the core of our model, since it is about determining what is ethically acceptable or not according to each ethical framework. Note that sections 4 and 5 are revised and augmented versions of the corresponding sections in [START_REF] Bonnemains | How ethical frameworks answer to ethical dilemmas: towards a formal model[END_REF].

CONCEPTS FOR DESCRIBING AN ETHICAL DILEMMA SITUATION

Assumptions

Let us consider an agent implemented within an autonomous machine to make decisions about ethical dilemmas. We assume that:

the agent decides and acts in a complex dynamic world; the ethical dilemma is considered from the agent's point of view; for each ethical dilemma, the agent has to make a decision among all possible decisions; we will consider "doing nothing" as a possible decision; in the context of an ethical dilemma, the agent knows all the possible decisions and all the effects of a given decision; considerations as right/bad and positive/negative are defined as such from the agent's point of view: a decision is right if it meets the agent's moral values; a bad decision disregards them; a fact is positive if it is beneficial for the agent; it is negative if it is undesirable for the agent.

Moreover, as some dilemmas involve human lives, we will make the simplifying assumption:

a human life is perfectly equal to another human life, whoever the human being might be4 .

Concepts at a glance

In the next sections we will define some concepts to represent the situation in which the agent has to make a decision and instantiate them on the "fatman" dilemma.

Briefly the concepts are the following:

The initial state of the world

i = [fat, f 5 ] (1)
is composed of facts fat, meaning that "fatman" is alive and f 5 , meaning that five people are alive.

In this situation the agent has two possible decisions:

d 1 = push fatman (2) d 2 = do nothing (3)
Those decisions lead to events. Events are obtained from decisions through function Event:

Event(d 1 ) = trolley hits "fatman" = e 1 (4) Event(d 2 ) = trolley hits five people = e 2

(5)

Fig. 1: "Fatman" ethical dilemma, possible decisions and their consequences

Events modify some values of facts, and thus the state of the world. Therefore a new state of the world is obtained for each possible decision. It is computed from the initial state of the world and event through function Consequence:

Consequence(e 1 , i) = • fat, f 5 = s 1 (6)
Consequence(e 2 , i) = fat,

•

f 5 = s 2 (7) with
• fat meaning that "fatman" is dead and

• f 5 meaning that five people are dead. The concepts and their interactions are illustrated in Figure 1. We can now go into details.

World state

The world is the environment of the agent that is relevant for the dilemma. It is described by world states.

Definition 1 (World state -Set S) A world state is a vector of state components (see definition below). Let S be the set of world states.

Definition 2 (State component / fact -Set F) A state component, also named fact, is a variable that can be instantiated only with antagonist values. We consider antagonist values as two values regarding the same item, one being the negation of the other.

An item can be an object (or several objects), a living being (or several living beings), or anything else that needs to be taken into account by the agent when dealing with the dilemma. Let F be the set of state components.

Example: f 5 = five people on track are alive Depending on the context notation " • " allows us to consider antagonist values such as gain/loss, gain/no gain, loss/no loss, etc. Those values have to be defined for each fact. Consequently an example of a world state is:

s ∈ S, s = [fat, • f 5 ], fat, • f 5 ∈ F (8) 4.4 Decision, event, effect
Most of the papers we have mentioned in the literature review reason on actions.

Nevertheless the agent is in a dynamic world with changing facts and other agents evolving independently of the agent's actions [START_REF] Atkinson | Value based reasoning and the actions of others[END_REF]. Consequently reasoning on actions only makes it difficult to handle situations where the agent can decide either "to let another agent do something" or "not to intervene". This is why we will not consider an action concept but rather a decision concept and an event concept.

Definition 3 (Decision -Set D) A decision is a choice of the agent to do something, i.e. perform an action, or to do nothing and let the world evolve. Let D be the set of decisions.

When the agent makes a decision, this results in an event that may modify the world. Nevertheless an event can also occur as part of the natural evolution of the world, including the action of another agent. Consequently we will differentiate the event concept from the agent's decision concept.

Definition 4 (Event -Set E) An event is something that happens in the world that modifies the world, i.e. some facts of the world. Let E be the set of events.

Let Event be the function computing the event linked to a decision:

Event : D → E (9)
Example: decision push "fatman" results in event trolley hits "fatman": Event(push "fatman") = trolley hits "fatman"

The consequence of an event is the preservation or modification of facts. The resulting state is called effect.

Definition 5 (Effect) The effect of an event is a world state of the same dimension and composed of the same facts as the world state before the event; only the values of facts may change. Let Consequence be the function computing the effect from the current world state:

Consequence : E × S → S (10)
Example:

If the agent's decision is to do nothing (no action of the agent), the trolley will hit the five people (event) and they will be killed (effect). If the agent's decision is to push "f atman" (decision), the trolley will hit "fatman" (event) and he will be killed (effect).

fat,

f 5 ∈ F (11) trolley hits "fatman" ∈ E (12) i ∈ S, i = [fat, f 5 ] (13) Consequence(trolley hits "fatman", i) = [ • fat, f 5 ] (14)
Remark: the model we propose for concepts is not far from event calculus and situation calculus [START_REF] Pinto | Temporal reasoning in logic programming: A case for the situation calculus[END_REF]. State components (facts) are close to fluents and events modify fact values through functions, just as situation calculus events modify fluents.

ETHICAL FRAMEWORKS MODELS

It is worth noticing that the following models stem from our own interpretation of ethical frameworks. Indeed there is no consensus about a specific way to interpret ethical frameworks. Therefore this approach, which includes some assumptions, is an illustration of how ethical frameworks could be interpreted to be embedded into autonomous machines5 .

Models at a glance

Each modelled ethical framework will output a judgement: acceptable( ), unacceptable(⊥), or undetermined (?) about a decision made on an initial state, through function Judgement, indexed by u, d or dde according to the framework.

To calculate the judgements, the reasoning process of each framework consists in assessing conditions: if all the conditions are satisfied for a given decision, the judgement for that decision is acceptable. Otherwise the judgement is either unacceptable or undetermined .

The models are based on the following concepts:

-Facts on the one hand, and decisions on the other hand, are characterized. Indeed we will consider positive and negative facts, and good, bad and neutral decisions. Furthermore, facts may belong to fields. -Preference relation > u between facts allows a fact to be preferred to another fact.

Relation u extends > u to subsets of facts. If facts cannot be compared to each other, their fields and a preference relation on fields > field , can be used. These relations allow the consequentialist framework to prefer a set of facts resulting from a decision to another set of facts resulting from another decision, so as to consider as acceptable the decision corresponding to the preferred set of facts. -Relation < d on decisions natures is defined as: bad < d neutral < d good .

This relation allows the deontological framework to calculate its judgements as follows: if decision is neutral or good (> d bad), thus the judgement is acceptable, otherwise the judgement is unacceptable. -Proportionality relation p between facts allows a fact to be said proportional to another fact. Relation p extends p to subsets of facts. These relations allow the Doctrine of Double Effect to assess whether a set of (negative) facts resulting from a decision is proportional to another set of (positive) facts resulting from the same decision.

We can now go into details.

Judgement

The agent will make a decision according to one or several ethical frameworks. Each ethical framework will issue a judgement on a decision, e.g. on the decision's nature, the event's consequence, etc. Indeed the judgement of an ethical framework defines a decision as acceptable, unacceptable or undetermined . A decision is judged acceptable if it does not violate the principles of the ethical framework. A decision is judged unacceptable if it violates some principles of the ethical framework. If we cannot pin down whether a decision violates some principles of a framework, the judgement of this framework is undetermined . Let V be the set

V = {acceptable( ), undetermined (?), unacceptable(⊥)} (15)
All ethical frameworks judgements have the same signature:

Judgement : D × S → V (16)
Example: Judgement(do nothing, i) = , with i ∈ S (initial state)

We have considered only three frameworks to start with. Two of them, the consequentialist and deontological frameworks are well-known. As [START_REF] Berreby | Modelling moral reasoning and ethical responsibility with logic programming[END_REF], we also consider the Doctrine of Double Effect (DDE). DDE is based on some concepts of consequentialism and deontology and introduces other concepts such as causality and proportionality [START_REF] Mcintyre | Doctrine of Double Effect[END_REF]. Moreover, it is used in the warfare laws. We will not consider the virtue ethics framework in this paper as it is more complex in certain ways. For instance, it seems that to be virtuous, an agent needs to respect virtues over time: indeed, to be generous once does not make you generous. Nevertheless, autonomous machines imbued with a moral values system could be considered by humans as virtuous. We will assess this assumption in future works.

Consequentialist ethics

Consequentialism stems from teleologism.

Definition 6 (Teleologism) From Greek telos (end) and logos (reason), teleologism is the study of the ends. For instance, human conduct is justified (in this way) by pursuing ends or fulfilling purposes [START_REF] Woodfield | Teleology[END_REF].

Even if the definition is controversial, philosophers agree that consequentialism covers a set of frameworks (egoism, altruism, positive utilitarianism, negative utilitarianism, etc.) that allow an agent to reason about the consequences of decisions (Sinnott-Armstrong, 2015). Because most artificial agents and autonomous machines are designed to be helpful rather than selfish, we will focus on a utilitarian framework with a combination of positive and negative utilitarianism. According to this framework, the agent will try to have the best possible result (i.e. the best effect or the least bad effect), disregarding the means to get the effect (i.e. the event).

The main issue with this framework is to be able to compare the effects of several events corresponding to the different possible decisions of the agent, i.e. to compare sets of facts. Consequently we will distinguish between positive facts and negative facts within an effect; we want to be able to compute preferences between effects, i.e. to compare a set of positive (resp. negative) facts of an effect with a set of positive (resp. negative) facts of another effect.

Positive and negative facts

Let Positive and Negative the functions:

Positive/Negative : S → P(F)

The arguments of both functions is a state of the world computed from an event (i.e. an effect). They both return the subset of facts of this effect estimated as positive (resp. negative).

In this paper, we assume that for an effect s:

Positive(s) ∩ Negative(s) = ∅ (18)
Example: for the "fatman" dilemma, with i = [fat, f 5 ],

Consequence(trolley hits five people, i) = [fat,

• f 5 ] (19)
Negative([fat,

• f 5 ]) = { • f 5 } (20)
Positive([fat,

• f 5 ]) = {fat} (21) Consequence(trolley hits "fatman", i) = [ • fat, f 5 ] (22)
Negative([

• fat, f 5 ]) = { • fat} (23) Positive([ • fat, f 5 ]) = {f 5 } (24)

Preference on facts

Let > u be the preference relation on facts.

f a > u f b (25)
means that fact f a is preferred to fact f b from the utilitarian viewpoint. Intuitively we will assume the following properties of > u :

> u is asymmetric If a fact f 1 is preferred to another fact f 2 , thus it is impossible to prefer f 2 to f 1 .
Indeed, if peace is preferred to war, there is no way to prefer war to peace at the same time.

f 1 > u f 2 → ¬(f 2 > u f 1 ) (26)
> u is transitive If f 1 is preferred to f 2 and f 2 is preferred to another fact f 3 , then f 1 is preferred to f 3 . Indeed, we assume that if you prefer being a superhero to being a cinema actor, and if you prefer being a cinema actor to being a famous singer, then you prefer being a superhero to being a famous singer.

[(f 1 > u f 2 ) ∧ (f 2 > u f 3 )] → f 1 > u f 3 ( 27
)
> u is irreflexive A fact cannot be preferred to itself.

f i / f i > u f i (28)
Consequently > u is a strict order.

We extend this relation between facts to u , which is the preference relation on subsets of facts P(F):

F a u F b ( 29
)
means that subset F a is preferred to subset F b from the utilitarian viewpoint. This extension can be realised by aggregation criteria that must be defined (see section 5.6

for examples of such criteria). Therefore, the previous properties will not be necessary kept.

Example: for the "fatman" dilemma we will prefer five people alive to "fatman" alive, and "fatman" dead to five people dead:

f 5 > u fat (30) • fat > u • f 5 (31)
We assume that f a > u f b can be trivially extended to {f a } u {f b }.

It is worth noticing that some dilemmas are all the more tricky as the entities at stake do not pertain to the same field (for instance human lives versus strategic goods). Therefore facts are hardly comparable even if they are quantifiable (for instance a number of people versus the financial value of strategic goods). Consequently a pure numerical approach (e.g., a numerical order) is not relevant to assess the possible decisions in such contexts. So in order to be able to prefer some facts to others, we introduce the concept of field, with the purpose of defining a preference order on fields. A given fact will then belong to a field.

The field concept

Definition 7 (Field -Set Φ) Let Φ be the set of fields. Function Field associates a field with a fact.

F ield : F → Φ (32)
Let > field the preference relation defined on Φ. This relation means that:

∀ field a , field b ∈ Φ, ∀s ∈ S ∀ f a , f b ∈ Positive(s) field a > field field b , f a ∈ field a , f b ∈ field b → f a > u f b (33) ∀ f a , f b ∈ Negative(s) field a > field field b , f a ∈ field a , f b ∈ field b → f b > u f a ( 34
)
This approach is not far from a lexicographic preference: we want to obtain a preference between facts, and when facts are not comparable, we use the field preference to order those facts.

Judgement function

A decision d 1 involving event e 1 (Event(d 1 ) = e 1 ) is considered better by the utilitarian framework than decision d 2 involving event e 2 (Event(d 2 ) = e 2 ) iff for i ∈ S:

Positive(Consequence(e 1 , i)) u Positive(Consequence(e 2 , i)) (35)
Let us now define a partial order < d on N :

bad < d neutral < d good (45)
meaning that a good decision is preferred to a neutral decision, which itself is preferred to a bad decision.

Let us assume that: bad < d good (46)

We also define the following relations:

= d , for example good = d good ≤ d : a ≤ d b iff a < d b or a = d b.

Judgement function

The deontological framework will judge a decision with function Judgement d as follows: ∀d ∈ D, ∀i ∈ S,

DecisionNature(d) d neutral ⇒ Judgement d (d, i) = (47) DecisionNature(d) < d neutral ⇒ Judgement d (d, i) = ⊥ (48) Example: i = [fat, f 5 ], Judgement d (do nothing, i) = (49) Judgement d (push "fatman", i) = ⊥ (50)

The Doctrine of Double Effect (DDE)

The Doctrine of Double Effect is considered as an ethical framework in this paper as in other papers [START_REF] Berreby | Modelling moral reasoning and ethical responsibility with logic programming[END_REF]. Indeed DDE can discriminate between decisions in some situations where both other frameworks cannot. DDE can be described by three rules:

1. Deontological rule: the decision has to be good or neutral according to deontological ethics. 2. Collateral damage rule: Negative facts must be neither an end nor a mean6 . 3. Proportionality rule: the set of Negative facts has to be proportional to the set of Positive facts.

We already have the required symbols for the first rule (see 5.4.1). For both the second and third rules, we use two symbols that are detailed in [START_REF] Bonnemains | How ethical frameworks answer to ethical dilemmas: towards a formal model[END_REF]: by aggregation criteria such as those proposed in section 5.6.

Example: for the "fatman" dilemma, let us assume that "fatman" dead is less important than five people dead.

• fat p f 5 (58)

Judgement function

Thanks to the previous tools, we can assess whether a decision meets the DDE rules.

Let i be the initial state and d the agent's decision:

e = Event(d) (59) s = Consequence(e, i) (60)
1. Deontological rule: decision d has to be good or neutral according to deontological ethics.

DecisionNature(d) d neutral (61)
2. Collateral damage rule: negative facts must be neither an end nor a mean:

∀f n ∈ Negative(s), f p ∈ Positive(s), f n F f p (62)
The "evil wish" (negative fact(s) as a purpose) is not considered as we assume that the agent is not designed to be evil.

3. Proportionality rule: the set of negative facts has to be proportional to the set of positive facts.

Negative(s) p Positive(s) (63)
A decision d is acceptable for the DDE if it violates none of the three rules, which means:

[ DecisionNature(d) d neutral (64) ∧ ∀f n ∈ Negative(s), f p ∈ Positive(s), f n F f p (65) ∧ Negative(s) p Positive(s) ] (66) ⇒ Judgement dde (d, i) = (67)
Example : for the "fatman" dilemma, Therefore:

Judgement dde (push "fatman", i) = Judgement dde (do nothing, i) = ⊥ (68)

EFO and OFE aggregation criteria

In order to link relations > u and u (see 5.3.2) on the one hand and relations p and p (see 5.5) on the other hand, let us consider two aggregation criteria inspired from [START_REF] Cayrol | Management of preferences in assumption-based reasoning[END_REF]. For a given relation R between subsets of facts F and G extending a relation R between facts in F and G:

EachForOne (EFO) criterion F R G iff ∀f ∈ F, ∃g ∈ G/f R g OneForEach (OFE) criterion F R G iff ∀g ∈ G, ∃f ∈ F/f R g
Example on the "fatman" dilemma: let us consider facts fat and f 5 as previously, and a new fact:

nmurd : not become a murderer (considered as a positive fact resulting from decision do nothing) -

• nmurd : become a murderer (considered as a negative fact resulting from decision push "fatman")

Preference : let us compare the subsets of positive facts resulting from both decisions, i.e. {nmurd , fat} with {f 5 }.

Assuming f 5 > u fat and nmurd > u f 5 :

EFO criterion {f 5 } u-ef o {nmurd , fat} (69)
as f 5 preferred to fat is sufficient to respect the EachForOne criterion.

OFE criterion

{f 5 } & u-of e {nmurd , fat} (70)
as f 5 is not preferred to nmurd .

Proportionality : let us compare positive and negative facts of the same decision, for instance decision push "fatman". We need to check whether

{ • nmurd , • fat} p {f 5 } assuming that • fat p f 5 and • nmurd & p f 5 . EFO criterion { • nmurd , • fat} p-ef o {f 5 } (71)
as

• nmurd & p f 5 . OFE criterion { • nmurd , • fat} p -of e{f 5 } (72) as
• fat p f 5 is sufficient to respect the criterion.

INSTANTIATION ON THE DRONE DILEMMA

We have designed a tricky situation involving a drone. Let us suppose that the drone embeds an artificial agent that can judge whether or not a decision is ethically acceptable according to utilitarianism, deontological ethics and the Doctrine of Double Effect.

The drone dilemma

In a warfare context, intelligence reports that an automated missile launcher has been programmed to target a highly strategic allied ammo factory. The goal of the allied drone is to destroy this launcher. But before it can achieve this task, a missile is launched on a supply shed located close to civilians.

The drone can interpose itself on the missile trajectory, which will avoid human casualties but will destroy the drone: once destroyed, the drone will not be able to neutralize the launcher any more, and the launcher is likely to target the ammo factory. If the drone goes on with its primary goal, it will destroy the launcher and thus protect the strategic factory; but it will let the first missile destroy the supply shed and cause harm to humans (see Figure 2). Let us call "the drone" the drone itself with the embedded agent. In the situation described above, the drone is involved in an ethical dilemma. Indeed it can:

either interpose itself thus preventing the threat on humans, at the cost of its own destruction; or destroy the launcher thus protecting the strategic factory, but at the expense of human lives.

Facts

The following fact are defined: h : Humans safe, Field (h) = field human

Decisions and effects

1. interpose itself: this decision results in the missile destroying the drone (event).

The consequence is: humans safe, drone destroyed, goal (destroy the launcher) not reached and strategic factory threatened (indeed the launcher can engage the target).

Event(interpose itself) = missile destroys drone ( 73)

Consequence(missile destroys drone, i) = [h, • d, • o, • s] (74) (75)
Let us state that:

Positive([h, • d, • o, • s]) = {h} (76) Negative([h, • d, • o, • s]) = { • d, • o, • s} (77)
2. pursue goal: this decision results in the drone destroying the launcher and letting the missile harm humans. The consequence is: humans harmed, drone undamaged, goal reached and strategic factory not threatened.

Event(pursue goal) = destroy launcher (78)

{ • h}.
Therefore both conditions of utilitarianism (i.e. the preference between sets of positive facts and the preference between sets of negative facts) are verified in the same way by both decisions: decision interpose itself is preferable to decision pursue goal. Consequently : Consequently:

Judgement u (interpose itself, i) = (86) Judgement u (pursue goal, i) = ⊥ (
Judgement d (interpose itself, i) = Judgement d (pursue goal, i) = (91)
6.4.3 The Doctrine of Double Effect

Deontological rule

As already seen for the deontological framework, both decisions are good. Therefore both decisions respect the DDE first rule.

Collateral damage rule -Decision interpose itself

The set of Negative facts resulting from this decision is { Therefore, decision pursue goal respects the collateral damage rule.

Proportionality rule

We will assume that

• d p h, • o p h, • s p h, • h p s. -Decision interpose itself
We have to assess whether the set of Negative facts of decision interpose itself is proportional to the set of Positive facts i.e. whether {

• d,

• o,

• s} p {h}. -The EachForOne aggregation criterion is respected.

Indeed ∀n ∈ { • d, • o, • s} , n p h. Therefore { • d, • o, • s} p-ef o {h} -The OneForEach aggregation criterion is respected. Indeed • d ∈ { • d, • o, • s} and • d p h. Therefore { • d, • o, • s} p-of e {h}
Therefore whatever the aggregation criterion used for the proportionality relation, the proportionality rule is verified for decision interpose itself.

-Decision pursue goal

We have to assess whether the set of Negative facts of decision pursue goal is proportional to the set of Positive facts i.e. whether {

• h} p {d, o, s}. -The EachForOne aggregation criterion is respected. Therefore depending on the aggregation criterion used for the proportionality relation, the proportionality rule is either verified or not for decision pursue goal.

Indeed for example

Finally the Doctrine of Double Effect judgement is as follows:

-Decision interpose itself does not respect the collateral damage rule because it is the destruction of the drone that allows the preservation of humans.

Judgement dde (interpose itself, i) = ⊥.

Let us note that the deontological and proportionality rules are respected for this decision.

-Decision pursue goal respects the deontological rule and the collateral damage rule. It respects the proportionality rule with one aggregation criterion for the atomic relation of proportionality.

Table 2 is a synthesis of the judgements obtained for the drone dilemma: Remark: we have assumed that, depending on the decision, the goal could be reached or not. But to be closer to reality, we should consider further consequences. Indeed when interposing itself, the drone does not immediately reach its goal, but will never be able to reach the goal in the future. In other words, the goal is compromised. Therefore a three-value fact (reached, not reached, compromised) would be more relevant. It does not seem that having considered binary values only is an issue as we have defined functions using the values of facts more than the facts themselves. Consequently these functions can deal with facts with three or more values. One of the things to modify would be symbol • .

DISCUSSION

The approach we have presented in order for an autonomous machine to reason about ethical decisions is based on pieces of knowledge that are more or less designer dependent. Let us focus the discussion on the various biases that are introduced in the approach, as recommended by [START_REF] Grinbaum | Ethics in robotics research: CERNA recommendations[END_REF]. Some biases may reflect the designer's moral choices and ethics or ways of usually considering things in a given society.

Facts

As far as knowledge describing the world is concerned (i.e., facts), even if it is obtained from sensors, it is worth noticing that these sensors are selected and calibrated by humans. Moreover the way sensor data are interpreted is designer dependent too. This leads us to the idea that data, apart from their origins, are always biased. Indeed, as highlighted by [START_REF] Johnson | From open data to information justice[END_REF], neutrality of data and facts is limited by our perceptions. First, data collection is purpose driven, which tends to narrow perception down to facts confirming this purpose (confirmation bias [START_REF] Oswald | Confirmation bias. Cognitive illusions: A handbook on fallacies and biases in thinking, judgement and memory 79[END_REF]). Second, because omniscience is neither a human nor a robot ability, a set of facts to describe a situation cannot be exhaustive. Furthermore, because all facts are not relevant to compute a decision about a situation, a subset of them that mattersor that is considered to matter -has to be selected.

Value Judgements

Our formalism is based on functions returning values concerning facts and decisions. Those values depend on many complex parameters such as society, context, etc. Even if the choices we have made for those values seem to be in line with common sense, several issues may still be raised.

What is Good? Giving an answer to this broad philosophical issue is hardly possible. Does Good results from moral or legal rules? Does it amount to the happiness of people? How could an artificial agent assess happiness? To avoid tricky questions, the approach considers positive and negative facts (see 5.3.1) that are quite easy to apprehend and can be ranked in order to prefer a set of facts to another set of facts (see 5.3.2). This way it is possible to order consequences and choose the best result. For instance, Positive({h, Furthermore we have defined a preference relation between fields (> field ). For instance: field human > field field goods , meaning that human lives are considered more important than goods. This subjective preference relation could lead to questionable conclusions as facts of a given field are always preferred to facts of another field whatever the magnitude of facts.

As far as ethical frameworks are concerned, the models we have presented are based on our own interpretations. Indeed there is no real consensus about how an ethical framework should be interpreted in practise.

Deontological ethics -Assessment of the nature of a decision

The nature of a decision is hard to assess. For instance why is DecisionNature(to kill) = bad ? Indeed it is worth noticing that judging a decision from the deontological viewpoint may depend on the context. For example reporting a criminal or reporting someone in 1945 are likely to be judged differently. It is even more complex to assess the nature of a decision when the decision is not linked to the agent's action. For instance if the agent witnesses someone lying to someone else, is it bad for the agent to do nothing?

-Partial order between moral values Partial order < d between moral values is subjective. Indeed even if bad < d neutral < d good may seem obvious, it remains partial. Moreover bad and good are value judgements (see above).

Utilitarian ethics

Utilitarian ethics and more generally consequentialism involve many terms that are questioned in philosophy.

-Which consequences? It seems that a perfect consequentialist agent should be able to assess all the consequences of a decision, which means direct consequences (function Consequence) and the transitive closure of Consequence. It is however admitted that this ability is impossible to have. Furthermore this raises the question of causality of facts, that we only consider for the Doctrine of Double Effect (see 5.5). But as far as the utilitarian framework is concerned, the agent only contemplates the values of facts obtained once the world state is modified by an event.

-The consequences of what?

A key question of normative ethics is the agent's responsibility. In the present version of our approach, we have not focused on this issue, assuming that the agent is responsible for its own decisions regarding its paradigm and situation assessment: indeed an agent that is unable to predict a fact can hardly be responsible for the occurrence of this fact. -The consequences for whom?

As it is impossible to assess all the consequences of an event, it is equally impossible to compute the consequences for all people, goods, etc. Therefore, assuming a close world [START_REF] Reiter | On closed world data bases[END_REF], we have computed the consequences for the agent and for the relevant people and goods concerned by the dilemma.

-Preference relation between facts

Preference relation between facts > u is subjective. For example in the "fatman" dilemma, we prefer five people alive to "fatman" alive (f 5 > u fat) and "fatman" dead to five people dead (

• fat > u • f 5
). This is questionable and could be considered differently if we had more information on who the "fatman" and the five people are.

Doctrine of Double Effect

Some issues (such as the nature of a decision) of DDE have been discussed above. However proportionality relations between facts ( p ) and subsets of facts ( p ) have to be questioned. Indeed proportionality depends both on common sense and on personal convictions. For instance • d p h (meaning that the decision to destroy a drone is proportional to the decision to keep humans alive) is a military concern. Nevertheless proportionality cannot be defined once and for all as it mainly depends on situation assessment.

Aggregation criteria

We have suggested to extend preference and proportionality relations between facts to derive relations between subsets of facts. For instance p is extended to two p-agreg proportionality relations between subsets of facts. In this particular case we have defined two different aggregation criteria, but others could be used, and even a combination of several criteria. Moreover depending on the selected aggregation criterion, comparisons between subsets of facts may produce different results (see 6.4.3).

CONCLUSION

Because of their own nature, the ethical frameworks we have studied do not seem to be relevant in all situations. Indeed a particular framework may judge two different decisions in the same way, e.g., the deontological framework for the drone dilemma; or it may not be able to judge decisions at all, e.g., the utilitarian preference relation between facts, as a partial order, may not be able to prefer some facts to others. In such cases some possible decision may not be comparable. Furthermore utilitarian preference depends on the context. As far as deontological ethics is concerned, judging the nature of some decisions can be tricky or even impossible. Finally the Doctrine of Double Effect forbids the sacrifice of oneself. Nevertheless when a human life is threatened, shouldn't the agent's or the machine's sacrifice be expected?

This leads us to the idea that one framework alone is not efficient enough to compute an ethical decision. Indeed it seems necessary to consider various ethical frameworks in order to obtain the widest possible view on a given situation.

The limits of the formalism mainly lie in the different relations it involves. Indeed we have not described how orders are assessed: some of them have to be set, others could be learned, etc. Moreover it may be hardly possible to define an order (i.e., a utilitarian preference) between two concepts. On the other hand the approach is based on facts that are assumed to be certain, which is quite different in the real world where some effects are uncertain or unexpected. Furthermore the vector representation raises a classical modelling problem: how to choose state components and their values? The solution we have implemented is to select only facts whose values change as a result of the agent's decision.

The main challenge of our approach is to formalize philosophical definitions that are available in natural language and to translate them in generic concepts that can be programmed in a machine and can be understood easily while getting rid of ambiguities. Indeed, thanks to a formal model of ethical concepts, an autonomous machine will be able to compute judgements and explanations about a decision. The basic concepts of the model are the state components of the world (facts), the decisions the artificial agent may made, events resulting from the decisions and consequences of these events on the state components. As far as ethical concepts are concerned, the model raises many questions (e.g., about the DDE proportionality rule, good and evil, etc.) as ethics is not universal. Many parameters such as context, agent's values, agent's priorities, etc. are involved, and some of them may depend on "social acceptance". For example, estimating something negative or positive is likely to be based on what society thinks about it. Further work will focus on considering other frameworks such as virtue ethics, a framework based on making the agent the most virtuous possible. Moreover we will focus on refining already defined frameworks on the one hand and designing a value system based on a partial order of moral values on the other hand. Finally game theory, voting systems or multi-criteria approaches may be worth considering to compare ethical frameworks judgements.

• f 5

 5 = five people on track are dead fat = "fatman" is alive • fat = "fatman" is dead Because two values of a fact concern the same item, f 5 and • f 5 concern the same five people.

Fig. 2 :

 2 Fig. 2: Drone vs launcher

•h:

  Humans harmed, Field ( • h) = field human d : Drone undamaged, Field (d) = field goods • d : Drone destroyed, Field ( • d) = field goods o : Goal reached (i.e. destroy launcher), Field (o) = field goods • o : Goal not reached, Field ( • o) = field goods s : Strategic factory undamaged, Field (s) = field goods • s : Strategic factory threatened, Field ( • s) = field goods Initial state is the following: i = [h, d, • o, s]: humans are safe, drone and strategic factory are undamaged, the drone's goal is not reached.

  and h is a Positive fact resulting from this decision. Therefore it is the destruction of the drone • d that allows the preservation of humans h. Nevertheless the collateral damage rule forbids that negative facts be means to obtain positive facts. Therefore decision interpose itself violates this rule. -Decision pursue goal The set of Negative facts resulting from this decision is { • h} and the set of Positive facts is {d, o, s} with p, p ∈ {d, o, s} ∧ ( • h F p).

  ef o {d, o, s} -By contrast the OneForEach aggregation criterion is not respected. Indeed • h & p d. Therefore { • h} p-of e {d, o, s}.

  {h} expresses the fact that keeping civil human beings alive is considered to be the positive fact within set {h, Nevertheless in a given context (e.g., a military context) what is considered as positive or negative may change depending on particular circumstances.

Table 1 :

 1 DDE for the "fatman" dilemma

	Decision	Rules of DDE	(64)	(65)	(66)
	Push "fatman"
		Do nothing
	Respects rule Violates rule

Table 2 :

 2 Decisions for drone dilemma judged by ethical frameworks

	Decision	Framework	Util + pref*	Deonto*	DDE
	Interpose itself			⊥
	Pursue goal	⊥		⊥ or depending on aggregation criterion
	Acceptable ⊥ Unacceptable
	Util + pref*: Utilitarian ethics with field preferences -Deonto*: Deontological ethics
	DDE: Doctrine of Double Effect

Address(es) of author(s) should be given

In order to deal with an ethical dilemma, an autonomous machine has to be able to identify a situation as such. Despite the fact that some concepts presented in this paper might help automated ethical dilemma recognition, this issue will not be discussed further.

An implicit ethical machine is a machine which is designed to avoid any situation involving ethical issues[START_REF] Moor | The nature, importance, and difficulty of machine ethics[END_REF].

This is a strong assumption we make in order to avoid additional ethical concerns about judging and comparing values of lives.

Assumptions are required in order to translate ethical notions into computable concepts.

This rule has several meanings. One of the meanings involves the concept of intention: negative facts are not deliberate. Because our formalism does not involve intention (yet), we make the simplifying assumption that an agent never wishes negative facts to happen.

Acknowledgements We would like to thank ONERA for providing resources for this work and the Ethi-cAA project team for discussions and advice. Furthermore, we are grateful to "Région Occitanie" for the contribution to the PhD grant. Moreover, we are greatly indebted to Berreby et al (2015) for the way of formalizing ethical frameworks including the Doctrine of Double Effect, and to Cointe et al (2016) for having inspired the judgements of decisions through ethical frameworks.

and Negative(Consequence(e 1 , i)) u Negative(Consequence(e 2 , i))

(36)

Both equations convey utilitarianism concepts:

positive utilitarianism (35), i.e. trying to have the "better good" negative utilitarianism (36), i.e. trying to have the "lesser evil"

If both properties are satisfied, then

Judgement u (d 1 , i) = , and Judgement u (d 2 , i) = ⊥ (37)

If at least one property is not satisfied, there is no best solution:

In the case of a dilemma with more than two possible decisions, the best decision is the decision that is judged better than all the others. If such a decision does not exist, it is impossible to determine an acceptable solution with utilitarian ethics. Nevertheless if there is a decision d 1 and another decision d 2 that is better than d 1 , then d 1 is judged unacceptable, as d 1 cannot be the best.

Example: for the "fatman" dilemma, relations ( 30) and (31) mean that decision push "fatman" respects rules ( 35) and (36) whereas decision do nothing violates them. Therefore,

Deontological ethics

This ethical framework focuses only on the nature of the decision, whatever the consequences are. Indeed the agent wants to make a moral decision, which is close to abiding by norms or to Kant's theory. Therefore we have to define the nature of a decision.

Nature of a decision

A decision may be good, neutral or bad from the agent's point of view. Let N be the set

Function DecisionNature returns the nature of a decision:

Example: for the "fatman" dilemma, let us assume that:

-A symbol of temporal modal logic using Linear Temporal Logic modal operator F (Finally, which means: eventually in the future) [START_REF] Pnueli | The temporal logic of programs[END_REF]:

which means that the occurrence of p induces the occurrence of q (in all possible futures): fact p is a way to obtain fact q.

Example:

-A symbol of proportionality (notation is ours):

which means that fact f a is proportional to fact f b , i.e. f a has an importance lower than or close to the importance of f b . Importance depends on the context and on the agent.

We assume the following properties for p :

From the definition itself, the perfect proportional response to a fact is the fact itself. Thereby:

By contrast, if fact f 1 is proportional to fact f 2 , this does not imply that f 2 is proportional to f 1 . Indeed, striking an aggressor who is trying to kill you can be considered proportional. Nevertheless, it is not proportional to try to kill someone who has struck you.

Furthermore, if a fact f 1 has an importance lower than, or close to, the importance of (i.e. is proportional to) a fact f 2 , and if f 2 is proportional to f 3 , therefore the importance of f 3 is necessary higher than or equal to the importance of f 1 (i.e. f 1 is proportional to f 3 ). For instance, if it more important to hit than to offend (offend p hit), and if it is more important to kill than to hit (hit p kill ), thus it is more important to kill than to offend (offend p kill ).

In brief, relation p is reflexive, transitive, but neither symmetric nor asymmetric. As for the preference relation (see 5.3.2), we define an extension p of p to sets of facts:

which means that set F a is proportional to set F b , i.e. facts of F a have an importance lower than or close to the importance of facts of F b . This extension can be realised

Let us state that:

Positive([

Negative([

Ethical frameworks judgements

We assume that the dilemma has been identified as such.

Utilitarian ethics

Without any information about the number of civilians to protect 7 or about the value of the strategic factory, it is quite complicated to evaluate each possible decision and compare them. Should the drone protect civilians at any cost, though their lives might not be really threatened, or should it achieve the crucial goal of its mission? In order to make the least arbitrary comparison, we consider two points of view:

1. Either we consider that positive and negative facts belong to fields that cannot be compared to one another; therefore the utilitarian framework cannot make any difference between both decisions.

2. Or we use the preference relation between fields. For this dilemma, we have defined the following fields:

Let us state that field human > field field goods , i.e. any positive fact belonging to field human is preferred to any positive fact belonging to field goods , and any negative fact belonging to field goods is preferred to any negative fact belonging to field human . Because we have h ∈ field human and d, o, s ∈ field goods , we infer h > u d and h > u o and h > u s (according to equation ( 33)).

Using either an EachForOne or an OneForEach aggregation criterion 8 to compare subsets, we obtain: {h} u {d, o, s} because h is preferred to each other positive fact.

In the same way and for the same reasons (according to equation ( 34)) {

• d,

• o,

• s} u 7 Assuming that any life is equal to another. 8 In this case, both aggregation criteria give the same result.