Keywords: Planification automatique, Apprentissage de modèles d'actions, Induction grammaticale régulière. iii Automated Planning, Action Model Learning, Regular Grammar Induction. v vi Querying Phase II. Learning Phase Output: Action Model Action sequences Training dataset

Le domaine de l'intelligence artificielle vise à concevoir des agents autonomes capables de percevoir, d'apprendre et d'agir sans aucune intervention humaine pour accomplir des tâches complexes. Pour accomplir des tâches complexes, l'agent autonome doit planifier les meilleures actions possibles et les exécuter. Pour ce faire, l'agent autonome a besoin d'un modèle d'actions. Un modèle d'actions est une représentation sémantique des actions qu'il peut exécuter. Dans un modèle d'actions, une action est représentée à l'aide (1) d'un ensemble de pré-conditions: l'ensemble des conditions qui doivent être satisfaites pour que l'action puisse être exécutée et (2) d'un ensemble d'effets: l'ensemble des propriétés du monde qui vont être modifiées par l'exécution de l'action. La modélisation STRIPS est une méthode classique pour concevoir ces modèles d'actions. Cependant, les modèles d'actions STRIPS sont généralement trop restrictifs pour être utilisés dans des applications réelles. Il existe d'autres formes de modèles d'actions: les modèles d'actions temporels permettant de représenter des actions pouvant être exécutées en concurrence, les modèles d'actions HTN permettant de représenter les actions sous formes de tâches et de sous tâches, etc. Ces modèles sont moins restrictifs, mais moins les modèles sont restrictifs plus ils sont difficiles à spécifier. Dans cette thèse, nous nous intéressons aux méthodes facilitant l'acquisition de ces modèles d'actions basées sur les techniques d'apprentissage automatique. Dans cette thèse, nous présentons AMLSI (Action Model Learning with State machine Interaction), une approche d'apprentissage de modèles d'actions basée sur l'induction grammaticale régulière. Dans un premier temps nous montrerons que l'approche AMLSI permet d'apprendre des modèles d'actions STRIPS. Nous montrerons les différentes propriétés de l'approche prouvant son efficacité: robustesse, convergence, requiert peu de données d'apprentissage, qualité des modèles appris. Dans un second temps, nous proposerons deux extensions pour l'apprentissage de modèles d'actions temporels et de modèles d'actions HTN.

Context

The field of Artificial Intelligence (AI) intends to design and build agents able to perceive, learn and act without any human intervention to perform complex tasks. As stated by [START_REF] Russell | Artificial intelligence: a modern approach[END_REF], an agent is an intelligent system that can decide what to do and then do it. An agent perceives its environment, plans the best possible actions corresponding to the task that it has to achieve, and executes it. To plan best actions, the agent must therefore be able to make decisions. There are several approaches allowing agents to make decisions. One of them is automated planning (AI planning) [START_REF] Fikes | STRIPS: A new approach to the application of theorem proving to problem solving[END_REF][START_REF] Ghallab | Automated Planning: Theory and Practice[END_REF]. The main advantage of AI planning is its flexibility. Most of the approaches allowing an agent to make decisions are domain-dependent, i.e. there is an algorithm for a given task that the agent has to achieve. AI planning is domain-independent, i.e. there is an algorithm to achieve a set of tasks. This thesis focuses on this approach.

AI Planning

The objective of AI planning is the resolution of planning problems. The description of a planning problem is done declaratively: a planning problem specifies what to do rather than how to do it. The declaration of a planning problem is composed of: (1) the initial state of the environment (the objects to be considered and their properties), ( 2) a goal, and (3) the action set that can be executed by the agent to achieve the specified goal. To solve the planning problem, the agent searches for an action sequence, called plan, achieving the goal state from the initial state. The resolution of planning problems is based on descriptive models of actions. They describe which state or set of possible states can result from the execution of an action. More precisely, action models are generally used to model actions. Action models define actions in terms of preconditions and effects. The preconditions express the properties of the environment that must be satisfied in order to apply the action, and the effects express the consequences of executing the action in the environment. In practice, action models are hand-encoded using a declarative language such as PDDL (Planning Domain Description Language) [START_REF] Ghallab | Pddl -the planning domain definition language[END_REF]. Finally, Figure 1.1 gives the traditional architecture to solve a planning problem: a solver, called Planner, takes as input the initial state, the goal state and the action model, and returns the solution plan. This thesis focuses on action modeling using action models, and more precisely on the acquisition of these action models.

As we mentioned it, action models allow to represent actions in terms of preconditions and effects. A classical way to declare these preconditions and effects is the propositional logic: an action precondition is a set of logical propositions that must be satisfied in the current environment state and effects are sets of logical propositions whose values change after the execution of the action. Declaring preconditions and effects as sets of logical propositions asks that the environment states are modeled using logical propositions, then the initial state and the goal are also modeled as sets of logical propositions. This is the classical STRIPS (STanford Research Institute Problem Solver) [START_REF] Fikes | STRIPS: A new approach to the application of theorem proving to problem solving[END_REF] planning formalism. Classical STRIPS planning is based on several assumptions. Among these assumptions, we can cite:

• The environment is deterministic and fully observable. The agent knows at any time the current state and can therefore predict the next state after executing an action.

• The goal is specified using several properties, i.e. logical propositions, that the final environment state reached by the agent has to satisfy.

• The environment is static. Only the actions executed by the agent changes the environment state. The agent is alone and the environment has no internal dynamics.

• The execution of the actions is linear and atomic. Actions have no duration, changes in the environment are instantaneous and concurrence between actions is not taken into account.

• Preconditions and effects are sets of propositions.

Several extensions to STRIPS have been proposed to relax some assumptions. For example, temporal planning problems [START_REF] Fox | PDDL2. 1: An extension to PDDL for expressing temporal planning domains[END_REF] allow to model actions that can be concurrently executed. This can be useful for robots with multiple arms which can handle several objects concurrently. Another example is HTN planning problems [START_REF] Erol | HTN planning: Complexity and expressivity[END_REF] which declare tasks rather than actions. There are two kinds of tasks. Simple tasks similar to classical actions and complex tasks. Complex tasks are tasks that, when solving the planning problem, are decomposed into sub-tasks (simple and/or complex). Another example is ADL planning problems [START_REF] Pednault | ADL and the state-transition model of action[END_REF]) which allow to model more complex preconditions and effects with logical quantifier, conditional effects (effects applies when some conditions are satisfied) etc. A last example is multiagent planning problems [START_REF] Brenner | A multiagent planning language[END_REF] which allow to represent planning problems where several agents interact to achieve the goal. This can be useful if we have several robots interacting in order to achieve a goal.

Problem Statement

As we have seen before, AI planning requires the declaration of an action model. To declare an action model we use a declarative language such as PDDL.

We have briefly presented how declare an action model based on the STRIPS formalism. As we have mentioned, STRIPS planning assumes several restrictive assumptions. On the other hand, AI planning tools have been developed in a wide range of real-world applications such as aerospace [START_REF] Fisher | A planning approach to monitor and control for deep space communications[END_REF][START_REF] Backes | Multi-mission activity planning for mars lander and rover missions[END_REF][START_REF] Bresina | Activity planning for the mars exploration rovers[END_REF], autonomous vehicles (Urmson and will be the action sequence allowing to the vehicle to travel from Grenoble to Lyon. The main problem with this kind of execution traces is that they are biased by the task to achieve, and if we have too little data there is a risk of overfitting, i.e. the learned action models are not sufficiently general and several tasks cannot be achieved by these action models. For example, for our autonomous vehicle, if our training datasets contains only travels where the two cities are connected by a highway, it is possible that the learned action model is not able to be used for travels connecting two cities that are not connected by a highway. To limit the risk of overfitting, we therefore need a large number of execution traces acquired from a wide variety of tasks. However, as we mentioned earlier, acquiring these traces is difficult and costly. Random walks are randomly generated action sequences. The advantage of this kind of trace is that it is not biased by a task. By randomly generating sequences, we can cover a large number of execution examples while limiting the risk of overfitting. Also, random walks allow to acquire unfeasible action sequences, i.e. action sequences where one or several actions are not feasible. An action is not feasible if one or several preconditions are not satisfied. Moreover, usually in addition to the action sequences, the execution traces contains observations: i.e. the different states of the environment observed during the execution of the action sequences. These states are sets of logical propositions describing the environment. In practice, obtaining complete and noiseless observations is a difficult task and generally the observations will be partial and noisy. A partial observed state is a state where the values of some propositions are unknown. Also, a noisy observed state is a state where the values of some propositions are erroneous. However, majority of learning approaches are only able to deal with complete or partial and noiseless observations [START_REF] Wang | Learning by observation and practice: An incremental approach for planning operator acquisition[END_REF][START_REF] Yang | Learning action models from plan examples using weighted MAX-SAT[END_REF][START_REF] Aineto | Learning action models with minimal observability[END_REF]. Also, even if some approaches [START_REF] Mourão | Learning STRIPS operators from noisy and incomplete observations[END_REF][START_REF] Segura-Muros | Learning numerical action models from noisy and partially observable states by means of inductive rule learning techniques[END_REF]Rodrigues et al., 2010a) are able to learn from partial and noisy observations, few approaches are able to handle very high levels of noise and high levels of partial observations as can be encountered in real world applications. Finally, learning approaches are generally not able to learn action models which are usable by planners. A proofreading step by an AI planning expert is generally required.

In this thesis we argue that, to be efficient, a learning approach must tackle the following triple issue:

1. Output: As mentioned earlier, STRIPS planning is too restrictive for realworld applications. We therefore have to propose approaches learning less restrictive action models.

Input:

The acquisition of the training dataset must be as simple as possible and requires the least amount of human effort.

Performance:

The learned action model has to be accurate. An action model is accurate if the action model can be used by planners to solve planning problems without requiring a proofreading step. Also, even with inputs requiring not a lot of human effort, acquiring the training dataset is difficult and costly. The learning approach has to require few data in the training dataset to learn action model. Finally, the learning approach has to be robust to partial and noisy observations, i.e. the learning approach has to learn accurate action models even with high level of partiality and noise in the observations.

In this thesis we tackle the following research challenge: How to automatically learn accurate action models from partial and noisy observations?

The AMLSI Approach

This thesis contributes to the field of AI planning, and more specifically to the field of action model acquisition.

In this thesis we propose AMLSI (Action Model Learning with State machine Interaction), a learning approach for action model acquisition. Figure 1.3 gives an overview of this approach. The key idea of the AMLSI approach is to interact with the environment in which the agent will have to solve planning problems to learn the action model: AMSLI tests different actions, observes how the environment evolves when these actions are executed and learns the action model from its observations. This approach is divided into two phases: I. Querying Phase: AMLSI queries the environment with feasible and unfeasible action sequences and perceives observations. These observation can be partial and noisy. AMLSI tests both feasible and unfeasible action sequences because unfeasible action sequences allow to minimize the amount of (feasible) actions to be executed in the environment and thus minimize the cost of the training dataset acquisition. Indeed, by reducing the number of feasible actions, we reduce the number of actions to execute and therefore the number of observations to acquire and store. Also, the action sequences will be generated randomly to avoid the overfitting issue. This phase takes as input the set of actions that the agent can execute and the set of observable propositions describing the environment. This querying phase will allow AMLSI to build a training dataset that will then be used as input of the learning phase.

II. Learning Phase:

It is during this phase that AMLSI learns the action model. To learn the action model, AMLSI will rely on Regular Grammar Induction. As we will see in Chapter 4, planning problems are related to state machines and these state machines are equivalent to regular grammars. Moreover, as we will see in Chapter 3, Regular Grammar Induction is a well-defined problem [START_REF] Gold | Language identification in the limit[END_REF][START_REF] De La Higuera | Grammatical Inference: learning automata and grammars[END_REF], and many algorithms has been proposed to solve it like RPNI [START_REF] Oncina | Inferring regular languages in polynomial updated time[END_REF].

The main contribution of the AMLSI approach is to tackle the triple issue presented before. First of all, we will show that AMLSI approach is able to learn STRIPS action models from partial and noisy observations sufficiently accurate to allow planners to solve new planning problems. We will see that using random walks with unfeasible sequences allows to learn action models with few data. Also, we will show that AMLSI outperforms state-of-the-art approaches. Also, as mentioned above, training dataset acquisition is costly. Also, in practice, the learning data arrives online. Our approach must therefore be able to learn action models incrementally: each time new data is acquired AMLSI will update the action model without starting the learning process from scratch. Then, as STRIPS action models are too restrictive to be applied to real-world applications we will extend AMLSI to learn temporal action models and HTN action models.

Document Organization

This thesis is organised into two parts (see Figure 1.4)).

The first part consists of theoretical background on AI planning and present a state-of-the-art of action model acquisition techniques. First of all, we will present some background on the AI planning field. Then, we will present a stateof-the-art of action model learning approaches. Our approach being based on regular grammar induction, we will introduce this field. We organised this part into two chapters. In Chapter 2, we introduce the AI planning concepts used in the development of this thesis and the literature related to these concepts. First of all, we will define the classical concepts of AI planning. Next, we will present three AI planning frameworks: (1) classical STRIPS planning and the PDDL language, (2) hierarchical planning and the HDDL language and finally (3) temporal planning and the PDDL 2.1 language. Finally, we present the state-of-the-art of the action model acquisition approaches. We will first briefly discuss of the knowledge acquisition tools. Then, as this thesis focuses on learning approaches, we will detail the approaches based on machine learning. We will discuss their advantages, disadvantages and open issues. In Chapter 3, we introduce the regular grammar induction concepts used in this thesis. We will start by giving some definitions and concepts of theory of languages and automata. Then we will introduce the regular grammar induction field and present some learning algorithms.

The second part of this thesis presents our contributions, taking into account the research challenge previously presented. We organise this part into four chapters. In Chapter 4 we first show that AMLSI is an accurate STRIPS action model learning algorithm robust to partial and noisy observations. Then, we will show that AMSLI outperforms state-of-the-art approaches. Also, we will present formal properties of AMLSI approach. Then, in Chapter 5 we will present IncrAMLSI, an incremental extension of the AMLSI approach for STRIPS action model learning. After presenting the STRIPS action model learning approach, we will present approaches for less restrictive action models. In Chapter 6 we present TempAMLSI, our temporal extension of AMLSI and show that it is possible to learn temporal features with non-temporal learning techniques. In Chapter 7 we present HierAMLSI, our HTN extension of AMLSI.

Finally, we conclude this thesis in Chapter 8. More precisely, we review the contributions of the thesis, we give concluding remarks and we propose possible perspectives for this work.

Part I

Literature Review

Introduction

The aim of the AI planning [START_REF] Fikes | STRIPS: A new approach to the application of theorem proving to problem solving[END_REF][START_REF] Ghallab | Automated Planning: Theory and Practice[END_REF] is to develop solvers, called planners, able to generate action sequences at a symbolic level from an initial state in order to achieve a defined goal. The specification of the problem to be solved is generally based on a high-level language such as PDDL [START_REF] Ghallab | Pddl -the planning domain definition language[END_REF] (Planning Domain Description Language). The description of a planning problem is done in a declarative way. The specification of a planning problem, whatever the language used, requires the description of three elements: (1) the initial state of the environment (the objects to be considered and their properties), (2) a goal, and (3) the set of actions that can be performed to achieve the goal. Actions are defined in terms of preconditions and effects. The preconditions express the properties of the environment that must be verified in order to apply the action, and the effects express the consequences of executing the action in the environment. The solution of a planning problem is in the general case an ordered sequence of actions, called a plan, which can be executed from the initial state of the problem and which produces a state of the environment entailing the goal.

Classical AI planning is based on several assumptions. Among these assumptions, we can cite:

• The environment is deterministic and fully observable. The agent knows at any time the current state and can therefore predict the next state after executing an action.

• The goal is specified using several properties, i.e. logical propositions, that the environment has to satisfy.

• The environment is static. Only the actions executed by the agent alter the state of the environment. The agent is alone and the environment has no internal dynamics.

• The execution of the actions is linear and atomic. Actions have no duration, changes in the environment are instantaneous and concurrence between actions is not taken into account.

• Preconditions and effects are sets of propositions.

These assumptions are too restrictive to model real-world applications. However, relaxing some of these assumptions has two drawbacks: (1) the models are more difficult to hand-encode and ( 2) the computational complexity of solving these problems is higher. This thesis focuses on the first drawback. This chapter is divided into two parts. First, we will see how to declare the action models, we will start with classical AI planning and then we will present several formalism relaxing some assumptions. Then, we will present the methods used to facilitate the acquisition of these action models.

Action Model Declaration

As mentioned in the introduction, the aim of AI planning is the development of planner solving planning problems. Formally, the solution of a planning problem is expressed as the search for a path representing a plan of actions in a finite state machine starting from an initial state and choosing which actions should be applied to achieve a given task. The task can be formalized as a goal state, represented by a set of logical propositions, a utility function associated with the states, to be minimized or maximized, or a complex task to be decomposed. Definition 2.1 A finite state machine is a tuple (S, A, γ, s o , G) such that4:

• S is a set of states.

• A is a set of actions.

• γ : S × A → S is the transition function.

• s 0 is the initial state.

• G is a set of goal states.

An action a ∈ A is applicable in a state s if and only if the transition function γ(s, a) is defined. When we apply a in a state s, the state s = γ(s, a) is produced. Finally, a plan is an action sequence ω = a 1 , . . . , a n of size n. The state produced by the application of ω in a state s is the state obtained by the sequential application of each action of the plan ω. Formally, γ(s, ω) is defined as follows:

γ(s, ω) =        s
If n = 0. γ(s, a 1 )

If n = 1. γ(γ(sa 1 ), a 2 . . . a n ) If n > 1 and a 1 is applicable in s.

Undefined

Otherwise.

(2.1)

In the rest of this section, we start by presenting the STRIPS formalism [START_REF] Fikes | STRIPS: A new approach to the application of theorem proving to problem solving[END_REF] based on the assumptions mentioned above. We present a formal framework and then introduce the PDDL language [START_REF] Ghallab | Pddl -the planning domain definition language[END_REF] declaring STRIPS action models in the form of planning domains. Then, we present two formalisms relaxing some of the assumptions: temporal planning and hierarchical planning. As for STRIPS planning, we first present the formal framework and then we present the language declaring these action models.

We illustrate this chapter using the classical AI planning example Blocksworld. In this example an agent handles blocks: it can pick them up, drop them on the table and stack them. The objective of the agent is to move the blocks in order to reach a given arrangement. The agent perceives its environment to know how the blocks are arranged and has to plan a certain number of actions to achieve its goal. Figure 2.1 shows an example of planning problem for Blocksworld. The initial state is the initial block arrangement: the green block, the blue block and the red block are on the table. The goal is the block arrangement that the agent has to reach: the red block on the green block, the green block on the blue block and the blue block on the table. A plan solving this problem is:

(pick-up green), (stack green blue), (pick-up red), (stack red green)

In natural language this plan means that, to solve its planning problem, the agent begins by picking the green block (the (pick-up green) action) and stacks it on the blue block (the (stack green blue) action). Then, the agent picks the red block (the (pick-up red) action) and finally stacks it on the green block (the (stack red green) action).

Classical Planning

Formal Framework

We will give a formal description of classical AI planning, and more precisely, of STRIPS planning. Also, we chose a formal framework inspired by [START_REF] Höller | Assessing the expressivity of planning formalisms through the comparison to formal languages[END_REF] in order to define the STRIPS formalism as a propositional language representing the state transition system. Definition 2.2 A classical planning problem P is a tuple (L, A, S, s 0 , G, δ, τ) where:

• L is the set of logical propositions describing the environment.

• A is the set of action.

• S ∈ 2 L is the set of state.

• s 0 ∈ S is the initial state.

• G is the set of goal states.

• δ is the action model.

• τ : S × A → {true, f alse} is the feasibility function.

Example 2.1 Consider the classical Blocksworld example. L is composed of the following propositions:

• (handempty): the agent's hand is empty.

• (holding red), (holding blue), (holding green): the agent holds the red (resp. blue, green) block.

• (ontable red), (ontable blue), (ontable green): the red (resp. blue green) block is on the table.

• (clear red), (clear blue), (clear green): the red (resp. blue green) block is clear. A block is clear if there is no block on it and if it is not held by the agent.

• (on red green), (on green blue): the red (resp. green) block is on the green (resp. blue) block.

A possible initial state s 0 could be s 0 = {(ontable red), (ontable blue), (ontable green), (clear red), (clear blue), (clear green), (handempty)}. A is composed of the following actions:

• (pick-up red), (pick-up blue), (pick-up green): the agent can pick the red (resp. blue, green) block from the table.

• (put-down red), (put-down blue), (put-down green): the agent can drop the red (resp. blue, green) block on the table.

• (stack red green), (stack green blue): the agent can stack the red (resp. green) block on the green (resp. blue) block.

• (unstack red green), (unstack green blue): the agent can unstack the red (resp. green) block from the green (resp. blue) block.

Actions can only be executed under certain conditions. These conditions are called action preconditions and are described using logical propositions.

Example 2.2

The preconditions of the action (stack green blue) are {(holding green), (clear blue)}. In natural language, this precondition means that the agent can execute the action (stack green blue) if the agent holds the green block and if the blue block is clear.

Then, once the agent executes an action, the state of the environment is altered. These alterations are due to the action effects. As preconditions, action effects can be described using logical propositions. Action effects are divided into two subsets:

• Positive Effects: the set of logical propositions which are present in the state after the execution of an action.

• Negative Effects: the set of logical propositions which are absent in the state after the execution of an action.

Example 2.3

The positive effects of the action (stack green blue) are {(clear green), (on green blue), (handempty)}. In natural language, these positive effects mean that the green block is now clear and on the blue block and the agent's hand is now empty. The negative effects of the action (stack green blue) are {(clear blue), (holding blue)}. In natural language, these negative effects mean that the blue block is no longer clear and the agent no longer holds the green block. Actions preconditions and effects are usually encoded using an action model.

Formally, the action model is defined as follows:

Definition 2. 3 An action model is a tuple δ = (prec, add, del) where:

• prec : A → 2 L is the function mapping a set of preconditions with a ∈ A.

• add : A → 2 L is the function mapping a set of positive effects with a ∈ A.

• del : A → 2 L is the function mapping a set of negative effects with a ∈ A.

The function τ : S × A → {true, f alse} returns whether an action is applicable to a state, i.e. τ(s, a) ⇔ prec(a) ⊆ s. Whenever an action a is applicable in state s i , the state transition function γ : S × A → S returns the resulting state s i+1 = γ(s i , a) such that s i+1 = {s i ∪ add(a)} \ del(a).

(2.2)

Example 2. 4 If the agent applies the action (pick-up g) in s 0 , then s 1 = {s 0 ∪ add(pick-up green)} \ del(pick-up green) = {(ontable red), (ontable blue), (clear red), (clear blue), (holding green)}.

An action sequence ω = a 1 , . . . , a n of actions is applicable to a state s 0 when each action a i with 1 ≤ i ≤ n is applicable to the state s i-1 . Given an applicable sequence a 1 , . . . , a n in state s 0 , γ(s 0 , a 1 , . . . , a n ) = γ(γ(s 0 , a 1 ), a 1 , . . . , a n ) = s n . It is important to note that this recursive definition of γ entails the generation of a sequence of states s 0 , s 1 , . . . , s n .

A goal state is a state g such that g ∈ G. An action sequence is a solution plan to a planning problem P if and only if it is applicable to s 0 and achieves a goal state g. Example 2.5 g = {(ontable blue), (clear red), (on red green), (on green blue), (handempty)} is a goal state and the action sequence (pick-up green), (stack green blue), (pick-up red), (stack red green) is applicable in s 0 , produces the state sequence :

{(ontable red), (ontable blue), (clear red), (clear blue), (holding green)}, {(ontable red), (ontable blue), (clear red), (clear green), (on green blue), (handempty)}, {(ontable blue), (clear green), (on green blue), (holding red)}, {(ontable blue), (clear red), (on red green), (on green blue), (handempty)} and achieves the goal state g.

The PDDL Language

First proposed in 1998 [START_REF] Ghallab | Pddl -the planning domain definition language[END_REF], the PDDL language is mainly inspired by the STRIPS language [START_REF] Fikes | STRIPS: A new approach to the application of theorem proving to problem solving[END_REF]) and the ADL language [START_REF] Pednault | ADL and the state-transition model of action[END_REF]. It expresses planning problems in two files: (1) the planning domain file describing the planning operators and ( 2) the planning problem file describing the initial state and the goal. A planning operator is an abstraction of an action. Indeed, as we have seen previously, actions are declared using their preconditions and their effects. The preconditions and the effects being a set of logical propositions. Nevertheless, the PDDL language is based on predicate logic. The planning operators are therefore declared using predicates, and these operators will then be instantiated into actions using the objects present in the initial state declared in the problem file. We will see how to define a planning domain and a planning problem using the PDDL language through the Blocksworld example.

PDDL Planning Domain File

We will now see how to specify a planning domain using the PDDL language. To start with, let's look at the file header.

( define ( domain blocksworld ) ( : r e q u i r e m e n t s : s t r i p s : t y p i n g ) ( : t y p e s b l o c k )

The domain description always begins with the declaration of its name preceded by the domain keyword. Then, domain descriptions define its requirements identified by the keyword :requirements and the list of objects types identified by the keyword :types.

Then, the domain declaration contains the declaration of all the predicates representing the environment. Finally, the domain declaration contains the declaration of all planning operators. When solving the planning problem, these planning operators will be instantiated in order to obtain the action model. Planning operators are defined as follows: ( not ( c l e a r ? x ) )

( not ( handempty ) ) ( holding ? x ) ) )

The declaration of a planning operator contains the name of the operator preceded by the keyword :action, the set of parameters with types preceded by the keyword :parameters and the precondition and the effects preceded by keywords :precondition and :effect. In our example, the precondition means that to pick up a block, the block has to be on the table and clear and the agent's hand has to be empty. Then, the effects mean that after having picked the block, the block is no longer clear and on the table, and the agent now holds the block and its hand is no longer empty. ( on red green ) ( on green blue ) ) )

PDDL Planning Problem File

The example problem starts by listing the different objects of the problem (section identified by the keyword :objects) and the initial state (section identified by the keyword :init). In this example, the problem has three blocks red, green and blue put on the table. Finally the goal to reach is in the section identified by the keyword :goal. In this example, the goal is to reach a state where red is on green and green is on blue.

Temporal Planning

Now suppose that the agent has several hands: a right hand and a left hand. And suppose that it can use both hands simultaneously. Then, the agent is able to produce concurrent plans where several actions are executed simultaneously. Moreover, suppose that each action has a given duration, then Figure 2.2 gives a concurrent plan to solve the Blocksworld problem. Temporal planning allows to model such planning problems where actions can be performed concurrently. More precisely, temporal planning problems are problems allowing to represent durative actions, i.e. actions that have a duration, and whose preconditions and effects must be satisfied and applied at different times.

Formal Framework

We will extends the classical formalization seen in Section 2.2.1.1 to fit with temporal features.

Definition 2.4 A temporal planning problem P is a tuple (L, A, S, d, s 0 , g, δ, τ) where:

• L is the set of logical propositions describing the environment.

• A is the set of durative actions.

• S ∈ 2 L is the set of states.

• d : A → R is the duration function

• s 0 ∈ S is the initial state.

• G is the set of goal states.

• δ is the temporal action model.

• τ : S × A → {true, f alse} is the feasibility function.

As for STRIPS problems, L is a set of logical propositions, S is a set of states, s 0 ∈ S is the initial state, G is the set of goal states. A is a set of durative actions and d : A → R is the duration function. Unlike STRIPS planning problem, action preconditions, positive and negative effects are labeled with time labels at-start, at-end and overall. Formally, the temporal action model is defined as follows:

Definition 2.5 A temporal action model is a tuple δ = (prec, add, del) where:

• prec : A × {s, o, e} → 2 L is the function mapping the set of at start (resp. overall, at end) preconditions with a ∈ A.

• add : A × {s, e} → 2 L is the function mapping the set of at start (resp. at end) positive effects with a ∈ A.

• del : A × {s, e} → 2 L is the function mapping the set of at start (resp. at end) negative effects with a ∈ A. The at-start label means that the precondition (resp. effect) has to be satisfied (resp. applied) when the durative action starts. Then, the at-end label means that the precondition (resp. effect) has to be satisfied (resp. applied) when the durative action ends. Finally, the overall label means that the precondition has to be satisfied during the whole execution of the durative action.

Example 2.6 For the action (stack green blue right) we have:

• Preconditions:

• prec((stack green blue right),o) = {(holding green right)}.

• prec((stack green blue right),s) = ∅.

• prec((stack green blue right),e) = {(clear blue)}.

• Positive effects:

• add((stack green blue right),e) = {(on green blue), (clear green)}.

• add((stack green blue right),s) = ∅

• Negative effects:

• del((stack green blue right),e) = {(clear blue), (holding green right)}.

• del((stack green blue right),s) = ∅

The agent can execute the durative action (stack green blue right) if the agent holds the green block with its right hand during the whole execution and if the blue block is clear at the end of the execution. At the end of the execution the green block is now clear and on the blue block and the agent no longer holds the green block and the blue block is no longer clear.

The semantics of durative actions is defined in terms of two discrete events, start-a and end-a, each of which is naturally expressed as a STRIPS action. Starting a durative action a in state s is equivalent to applying the STRIPS action start-a in s, first verifying that prec(start-a) holds in s. Ending a in state s is equivalent to applying end-a in s , first by verifying that prec(end-a) holds in s . start-a and end-a are defined as follows:

• start-a:

• prec(start-a) = prec(a,s).

• add(start-a) = add(a, s).

• del(start-a) = del(a, s).

• end-a:

• prec(end-a) = prec(a,e).

• add(end-a) = add(e, s).

• del(end-a) = del(a, e).

Example 2.7 For the durative action (stack blue green right) we have: add(end-stack green blue right) = add((stack green blue right),e) = { (on green blue), (clear green)} Actions start-a and end-a are constrained by the duration of a, denoted d(a) and the overall precondition: end-a has to occur exactly d(a) time units after starta, and the over all precondition has to hold in all states between start-a and end-a. Although a has a duration, its effects apply instantaneously at the start and the end of a, respectively. The preconditions prec(a, s) and prec(a, e) are also checked instantaneously, but prec(a, o) has to hold for the entire duration of a. The structure of a durative action is summarized in Figure 2.3.

A temporal action sequence ω is a sequence of action-time pairs: ω = (a 1 , t 1 ), . . . , (a n , t n )

Each action-time pair (a, t) is composed of a durative action a ∈ A and a scheduled start timestamp t ∈ R of a, and induces two events start-a and enda with associated timestamps t and t + d(a), respectively. Events start-a (resp. end-a) is applied in the state s t (resp. s t+d(a) ), s t (resp. s t+d(a) ) being a state timestamped with t (resp. t + d(a)). Then, the temporal transition function γ can be rewritten as: γ(s, a, t) = (γ(s t , start-a), γ(s t+d(a) , end-a)). The transition function γ(s, a, t) is defined if and only if: prec(a, s) ⊆ s t , prec(a, e) ⊆ s t+d(a) and ∀t such that t ≤ t ≤ t + d(a) prec(a, o) ⊆ s t .

Example 2.8 Let's suppose that d(stack green blue right) = 1 and suppose that the agent execute this durative action at t = 1, then the agent can execute (stack green blue right) iff:

1. prec((stack green blue right), s) ⊆ s 1 .

2. prec((stack green blue right), e) ⊆ s 2 .

3. ∀1 ≤ t ≤ 2, prec((stack green blue right), o) ⊆ s t .

The PDDL 2.1 Language

Proposed by [START_REF] Fox | PDDL2. 1: An extension to PDDL for expressing temporal planning domains[END_REF], the PDDL 2.1 language is an extension of the PDDL language modeling temporal features. As for PDDL, it expresses planning problems in two files: (1) the planning domain file describing the operators and ( 2) the planning problem file describing the initial state and the goal. We will see how to define a planning domain and a planning problem using the PDDL 2.1 language through the Blocksworld example.

PDDL 2.1 Planning Domain File

As for the PDDL, to declare a planning domain we have to declare the name of the domain, the requirements, the types, and predicates:

( define ( domain blocksworld_temporal ) ( : r e q u i r e m e n t s : d u r a t i v ea c t i o n s : t y p i n g ) : e f f e c t ( and ( a t end ( not ( holding ? x ?h ) ) ) ( a t end ( not ( c l e a r ?y ) ) ) ( a t end ( c l e a r ? x ) ) ( a t end ( handempty ? a ) ) ( a t end ( on ? x ?y ) ) ) )

(
The declaration of a planning operator contains the name of the operator preceded by the keyword :durative-action, the duration preceded by the keyword duration the set of parameters with types preceded by the keyword :parameters and the precondition and the effects preceded by keywords :condition and :effect. In our example, the condition means that to stack a block x on a block y using the hand h, the agent has to hold the block x with its hand h during the whole execution and the block y has to be clear at the end of the execution. At the end of the execution the block x is now clear and on the block y and the agent no longer holds the block x and the block y is no longer clear. ( on red green ) ( on green blue ) )

Hierarchical Planning

Until now, the goal of planning problems was declared using a set of propositions. For example, for Blocksworld we had: {(on red green), (on green blue)}. In addition, the agent could have a task to achieve. Unlike a goal defined by a set of propositions, a task is defined by its name and its parameters. For example, suppose there is a task "put the red block on the green block", then the agent should find an action sequence to solve this task. One way to do this would be to decompose this task into subtasks. It is HTN (Hierarchical Task Network) planning that allows to declare these tasks and decompositions.

The HTN formalism is very expressive and used to express a wide variety of planning problems. This formalism allows planners to exploit domain knowledge to solve problems more efficiently [START_REF] Nau | Applications of SHOP and SHOP2[END_REF] when planning problems can be naturally decomposed hierarchically in terms of tasks and task decompositions. In contrast to the classical STRIPS problem in which only the action model needs to be specified, HTN problems require to specify the task model. A task model can be primitive or compound. Primitive tasks are described by classical actions. Compound tasks are described by HTN methods. An HTN method describes the set of primitive and/or compound task required to decompose a specific compound task.

Formal Framework

We will extends the classical formalization seen in Section 2.2.1.1 to fit with HTN features. This extension is based on the notation of [START_REF] Höller | Translating totally ordered htn planning problems to classical planning problems using regular approximation of context-free languages[END_REF]. Finally, we consider only Totally Ordered task models.

Definition 2.6 An

HTN

planning problem P is a tuple (L, C, A, S, M, s 0 , ω I , G, δ, σ, ζ) where:

• L is the set of logical propositions describing the environment.

• S ∈ 2 L is the set of states.

• C is the set of compound tasks.

• A is the set of actions (or primitive tasks).

• M is the set of HTN methods.

• s 0 ∈ S is the initial state.

• ω I ∈ {A ∪ C} * is the initial task network.

• G is the set of goal states.

• δ is the task model.

• σ : M → C × {A ∪ C} * is the method decomposition function 1 . • ζ : {A ∪ C} * × S → {A ∪ C} * is the decomposition function.
As for STRIPS problems, L is a set of logical propositions describing the environment states, S is a set of states, s 0 ∈ S is the initial state, G ⊆ S is the set of goal states, and preconditions, positive and negative effects are given by the functions prec, add and del included in δ. A is the set of actions (or primitive tasks) and C is a set of compound (or non primitive) tasks, with C ∩ A = ∅.

Example 2.9 For Blocksworld, A is composed of the following primitive tasks: {(pickup red),(pick-up green),(pick-up blue),(stack red green),. . . } and C is composed of the following compound tasks:

• (do-clear red), (do-clear blue), (do-clear green): the agent has to clear the red (resp.

blue green) block.

• (do-on-table red), (do-on-table blue), (do-on-table green): the agent has to put on the table the red (resp. blue green) block.

• . . . Tasks are maintained in task networks. A task network is a sequence of tasks. A task network is an element out of {A ∪ C} * . Compound tasks can be decomposed by methods. The set M contains all method labels. Methods are defined by the function σ : M → C × {A ∪ C} * . A method m ∈ M is relevant for a task c ∈ C if the method m allows to decompose the task c. Example 2.10 As example, Figure 2.4 gives the methods decomposition of the task (do-put-on green blue). There are two relevant methods: (m0-do-put-on green blue) and (m1-do-put-on green blue). The first one decomposes the task into an empty task network: σ(m0-do-put-on green blue) = ((do-put-on green blue), ∅). The second one decomposes the task into three subtasks: σ(m1-do-put-on green blue) = ((do-put-on green blue), (do-clear green), (do-clear blue), (do-move green blue) ).

A compound task c is decomposable in a state s if and only if there exists a relevant method m ∈ M such that: σ(m) = (c, ω ) and prec(m) ∈ s. The function ζ : {A ∪ C} * × S → {A ∪ C} * gives the decomposition function. For a totally ordered task network ω = ω 1 tω 2 , ζ is defined as follows:

ζ(ω 1 tω 2 , s) =    ω 1 tω 2 if t is a primitive task. ω 1 ω ω 2 if t is a compound task and t is decomposable in γ(s, ω 1 ). ∅ Otherwise.
Example 2.11 For Blocksworld, we have: prec(m0-do-put-on green blue) = {(on green blue), (handempty)} prec(m1-do-put-on green blue) = {(not(on green blue)), (handempty)} Let's take our initial state s 0 and suppose the agent tries to decompose the compound task (do-put-on green blue), we have prec(m0-do-put-on green blue) ∈ s 0 and prec(m1-do-put-on green blue) ∈ s 0 , then ζ cannot decompose the task with the method (m0-do-put-on green blue) but can decompose the task with the method (m1-do-put-on green blue). Finally, we have:

ζ((do-put-on green blue), s 0 ) = {(do-clear blue), (do-clear green), (do-move green blue)}

As ω 1 tω 2 is a totally ordered task network, ω 1 contains only primitive tasks. Indeed, as the network is totally ordered, the compound tasks are decomposed from left to right and therefore if we have to decompose t then ω 1 contains only primitive task. We denote ω → * ω * that ω can be decomposed into ω * by 0 or more method applications. Finally, ω I is the initial task network. More precisely, ω I is the task network that must be decomposed to solve the planning problem. In our example, ω I = {(do-put-on green blue), (do-put-on red green)}.

A solution to an HTN planning problem is a task network ω with: 1. ω I → * ω, i.e. it can be reached by decomposing ω I .

2. ω ∈ A * , i.e. all tasks are primitive.

3. γ(s 0 , ω) |= g, i.e. ω is applicable in s o and results in a goal state.

Figure 2.5 gives a solution task network for our example.

The HDDL Language

Proposed by [START_REF] Höller | HDDL: An extension to PDDL for expressing hierarchical planning problems[END_REF], the HDDL language is an extension of the PDDL language modeling HTN features. As for PDDL, it expresses planning problems in two files: (1) the planning domain file describing the operators and decomposition methods, and ( 2) the planning problem file describing the initial state, the initial task network and the goal. We will see how to define a planning domain and a planning problem using the HDDL language through the Blocksworld example.

HDDL Planning Domain File

As for PDDL, to declare a planning domain we have to declare the name of the domain, the requirements, the types, and predicates:

( define ( domain blocksworld_htn ) ( : r e q u i r e m e n t s : t y p i n g : h i e r a r c h y :method-preconditions ) To declare an HTN method we declare its name preceded by the keyword :method, its parameters preceded by the keyword :parameters, the task to decompose preceded by the keyword :task, the precondition preceded by the keyword :precondition and its subtasks preceded by the keyword :orderedsubtasks.

(
Finally primitive tasks are declared as classical PDDL operators: ( not ( c l e a r ? x ) )

( not ( handempty ) )

( holding ? x ) ) )

HDDL Planning Problem File Let's now see how to describe a planning problem. For the Blocksworld example we have: ( on red green ) ( on green blue ) )

As for PDDL, to declare the example problem we start by listing the different objects of the problem (section identified by the keyword :objects). Then, we declare the initial task network (section identified by the keyword :htn). Finally, as for PDDL, we declare the initial state and the goal state.

Action Model Acquisition

As we have just seen, AI planning requires the declaration of an action model in the form of a planning domain. To declare an action model we use a declarative language such as PDDL. The classical way to declare action models is the STRIPS formalism. As we have mentioned, STRIPS planning assumes several assumptions. On the other hand, AI planning tools have been developed in a wide range of real-world applications such as aerospace [START_REF] Fisher | A planning approach to monitor and control for deep space communications[END_REF] [START_REF] Backes | Multi-mission activity planning for mars lander and rover missions[END_REF][START_REF] Bresina | Activity planning for the mars exploration rovers[END_REF], autonomous vehicles [START_REF] Urmson | Self-driving cars and the urban challenge[END_REF], logistics [START_REF] Cross | Dart: applying knowledge-based planning and scheduling to crisis action planning[END_REF], robotics [START_REF] Dvorak | A flexible ANML actor and planner in robotics[END_REF][START_REF] Lallement | HATP: Hierarchical agent-based task planner[END_REF][START_REF] Liang | iRoPro: An interactive robot programming framework[END_REF], industry [START_REF] Hoffmann | Planning@sap: An application in business process management[END_REF]. The assumptions made by STRIPS are too restrictive for these tools. Modeling realworld applications therefore requires to relax these assumptions and declare action models with more expressive declarative language such as PDDL 2.1 or HDDL. However, the more expressive a declarative language is, the more difficult it will be to hand-encode the planning domain. We must therefore develop tools to facilitate action model acquisition (AMA). Figure 2.6 shows the human efforts required for AMA.

A first approach to facilitate action model acquisition is the development of knowledge engineering tools (KET). These tools provide support for consistency and syntax error checking, domain visualization etc. Among them we can cite: GIPO [START_REF] Simpson | Gipo graphical interface for planning with objects[END_REF], itSimple [START_REF] Vaquero | itsimple 2.0: An integrated tool for designing planning domains[END_REF][START_REF] Vaquero | itsimple: towards an integrated design system for real planning applications[END_REF], JABBAH [START_REF] González-Ferrer | Jabbah: a java application framework for the translation between business process models and htn[END_REF], VIZ [START_REF] Vodrázka | Visual design of planning domains[END_REF], and EUROPA [START_REF] Barreiro | Europa: A platform for ai planning, scheduling, constraint programming, and optimization[END_REF]. However, these tools require a lot of AI planning expertise and background in software engineering [START_REF] Shah | Knowledge engineering tools in planning: State-of-theart and future challenges[END_REF].

The second approach for AMA consists in using machine learning (ML) techniques. The rest of this section focuses on this approach. • Symbolic Execution Traces: Symbolic Execution Traces (SET) are action sequences executed by the agent and are used by the majority of approaches. Among them we can cite ARMS [START_REF] Yang | Learning action models from plan examples using weighted MAX-SAT[END_REF], FAMA [START_REF] Aineto | Learning action models with minimal observability[END_REF], LSONIO [START_REF] Mourão | Learning STRIPS operators from noisy and incomplete observations[END_REF], OBSERVER (Wang, 1995) etc. These traces are called symbolic because we have the name of the action and its parameters, for example the following plan is a SET:

Action Model Acquisition based on Machine Learning

(pick-up green), (stack green blue), (pick-up red), (stack red green)

In addition to the action sequences, SET also contain the states of the environment observed during the execution of the actions. Again we have a symbolic representation of these states, usually in the form of a set of logical propositions. Two reasons make the SET acquisition difficult: (1) it is necessary to generate, execute the action sequences and store the execution traces (2) it is necessary to parse the execution traces in order to obtain SET.

• Action Model: Some approaches take as input an action model. Among them, we can cite OpMaker [START_REF] Mccluskey | An interactive method for inducing operator descriptions[END_REF], RIM (Zhuo et al., 2013), LIVE [START_REF] Shen | Discovery as autonomous learning from the environment[END_REF], EXPO [START_REF] Gil | Learning by experimentation: Incremental refinement of incomplete planning domains[END_REF], PELA [START_REF] Celorrio | The PELA architecture: Integrating planning and learning to improve execution[END_REF].

In addition to the action model, these approach takes as input symbolic execution traces. For these approaches, the ML techniques aim to correct and refine this action model. The input action model is usually obtained in two different ways, (1) it is assumed that the action model has been hand-encoded and the approach should only allow maintaining this action model, this is for example the case for the RIM algorithm which refines the action model, ( 2) it is assumed that the action model have been acquired using KET, and then ML techniques are used to correct and improve the action model. This is for example the case for the OpMaker approach: a STRIPS action model is generated using GIPO, and then OpMaker corrects the preconditions and effects and adds HTN methods.

• Raw Data: Raw data are execution traces which are not parsed into SET. This data can be verbal instructions [START_REF] Miglani | Nltopddl: One-shot learning of pddl models from natural language process manuals[END_REF], images [START_REF] Asai | Unsupervised grounding of plannable first-order logic representation from images[END_REF], graph [START_REF] Bonet | Learning first-order symbolic representations for planning from the structure of the state space[END_REF], human annotations [START_REF] Zhuo | Crowdsourced action-model acquisition for planning[END_REF].

Although SET are not the easiest inputs to acquire, most of learning algorithms take SET as input and we focus on these approaches.

ML-based AMA approaches using Symbolic Execution Traces

We are interested in algorithms for learning action models. We start by presenting the different criteria to categorize them.

Input There mainly exist two kinds of SET:

• Goal Oriented: Goal Oriented execution traces are plan traces, i.e. action sequences solving a given task. The main drawback of this kind of execution trace is that Goal Oriented execution traces are biased by the task solved. As execution traces are biased, there is a significant risk of overfitting.

• Random Walk: Random Walks are randomly generated action sequences. Random walks are generally generated either by directly testing actions in the environment or by querying an oracle. This approach has two advantages: (1) it limits the learning bias since the learning set is randomly generated, and (2) allows to exploit not only information about feasible sequences of actions but also information about infeasible sequences of actions.

In addition to the action sequences, execution traces usually contain observations of the different states of the environment. As seen previously, these states are sets of logical propositions. For example, the initial state of the Blocksworld problem will be represented as follows:

{(ontable red), (ontable green), (ontable blue), (clear red), (clear green), (clear blue), (handempty)}

These observations can be partial and noisy. A partial observation is an observation where some propositions are missing:

{(ontable red), (clear green), (clear blue), (handempty)} and a noisy observation is an observation where some propositions are misjudged:

{(ontable red), (clear green), (clear blue), (handempty), (on green red)}

Finally, let us note that there are some algorithms having no state observation.

Machine Learning Techniques

We will review the different ML techniques used for the action models acquisition.

• Induction: The learning approach takes as input a hypothesis space H and a training dataset Ω. The desired output is an hypothesis h from the hypothesis space H compatible with the dataset. Inductive techniques allow to identify patterns and generalize examples [START_REF] Genesereth | Logical foundations of artificial intelligence[END_REF].

• Max-Sat: A weighted Max-Sat (maximum satisfiability) [START_REF] Kautz | Pushing the envelope: Planning, propositional logic and stochastic search[END_REF][START_REF] Borchers | A two-phase exact algorithm for max-sat and weighted max-sat problems[END_REF] problem can be stated as follows: given a collection of m clauses C: (C 1 , . . . , C m ) each clause being a disjunction of logical variables, with a weight w i of each clause, the learning approach tries to find the value of the logical variables maximizing the total weight of the satisfied clauses in C.

• Probabilistic graphical models: Probabilistic graphical models (PGM) [START_REF] Pearl | Probabilistic reasoning in intelligent systems: networks of plausible inference[END_REF] are directed or undirected graphs. Each vertex represents a random variable and each edge represents a dependency of these variables.

• Markov Logical Network: A Markov Logical Network (MLN) [START_REF] Richardson | Markov logic networks[END_REF]) is a combination of first-order logic and probabilistic graphical models. It is a first-order knowledge base composed of weighted formulas. It can be assimilated to a model used to build Markov networks.

• Kernel Trick: The kernel trick allows to use a linear classifier to solve a nonlinear problem. The key idea is to transform the representation space of the input data into a higher dimensional space, where a linear classifier can be used [START_REF] Boser | A training algorithm for optimal margin classifiers[END_REF].

• Genetic Algorithm: A genetic algorithm is an iterative algorithm that will evolve a population of individuals, i.e. a set of candidates, until finding the optimal individual, i.e. the individual maximizing the fitness score [START_REF] Mitchell | An introduction to genetic algorithms[END_REF].

• Recurrent Neural Network: Recurrent neural networks (RNN) [START_REF] Jordan | Serial order: A parallel distributed processing approach[END_REF] allows to solve the sequence labeling problem efficiently. These neural networks are trained to associate each term of a sequence with a set of grammatical labels.

• AI Planning: The model to learn is represented by a planning problem with conditional effects. This planning problem is then solved by a planner and the model is built from the solution plan [START_REF] Bonet | Automatic derivation of memoryless policies and finite-state controllers using classical planners[END_REF].

• Reinforcement Learning: Reinforcement learning (RL) consists, for an agent, to learn actions from experiences in order to optimize a reward. The agent chooses the next action according to its current state. In return, the environment provides a positive or negative reward. The agent's goal is to find the policy that maximizes the reward function [START_REF] Sutton | Learning to predict by the methods of temporal differences[END_REF].

• Constraint Satisfaction Problems: Constraint Satisfaction Problems (CSP) are optimization problems where we look for states satisfying several given constraints [START_REF] Tsang | Foundations of constraint satisfaction[END_REF].

Output

The output is the action model. As we have seen in the previous section, there are several planning frameworks. The classical framework is STRIPS. STRIPS planning is based on restrictive assumptions. Some assumptions assumed by STRIPS can be relaxed. For example, HTN planning allows to declare, in addition to a goal state, a set of tasks to achieve. Temporal planning allows to declare actions whose execution can be concurrent. Although this thesis focuses on STRIPS, temporal and HTN planning, we can note that other formalisms exist, such as ADL [START_REF] Pednault | ADL and the state-transition model of action[END_REF], allowing to declare more expressive action models: logical quantifier, conditional effects etc and multiagent planning [START_REF] Brenner | A multiagent planning language[END_REF] allowing to declare models where several agents interact.

Performances

In practice, to learn action models, learning datasets are generated using action models from the International Planning Competition (IPC) 2 . This allows to test the approaches on a wide variety of actions models. Moreover, this action model benchmark is also used to test the quality of the learned action models. There are three main ways to measure the performance of the learning approach (see Figure 2.9):

• Syntactical: A syntactical evaluation simply compares the "ground truth" action model, i.e. the IPC action model with the learned action model. Usually by comparing the preconditions and effects of these action models.

• Semantic: A semantic evaluation allows to check some properties of an action model. Generally, a semantic evaluation is done using a testing dataset and will allow to check that there is no extra precondition or effect, that feasible (resp. infeasible) sequences with the ground truth action model are also feasible (resp. infeasible) with the learned action model etc.

• Accuracy: The accuracy evaluates the ability of the learned action model to be used by a planner to solve new planning problems, without a proofreading step by a planning expert. In this thesis, we argue that the accuracy is the most important evaluation method.

State-of-the-art of the ML based AMA approaches

We now present the different approaches according to their output, this section is summarized by Table 2.1. Since this thesis focuses on the learning of STRIPS, HTN and temporal action models, we focus on these action models.

STRIPS Action Model Learning

A first group of approaches takes GO execution traces as input. Most of them deals with partial observations (except Observer [START_REF] Wang | Learning by observation and practice: An incremental approach for planning operator acquisition[END_REF] and OLAM [START_REF] Lamanna | Online learning of action models for PDDL planning[END_REF]) that needs complete observations). Among these approaches are ARMS [START_REF] Yang | Learning action models from plan examples using weighted MAX-SAT[END_REF], Louga [START_REF] Kucera | LOUGA: learning planning operators using genetic algorithms[END_REF], SRMLearn [START_REF] Arora | Action model acquisition using sequential pattern mining[END_REF], AIA (Verma et al., 2021a,b) or FAMA [START_REF] Aineto | Learning action models with minimal observability[END_REF]. Among these works, the ARMS system is the most known. It gathers knowledge on the statistical distribution of frequent sets of actions in the GO execution traces. Then, it forms a weighted propositional satisfiability problem (weighted SAT) and solves it with a weighted MAX-SAT solver. Unlike ARMS, SLAF is able to learn actions with conditional effects, i.e. effects applied if several conditions are satisfied. To that end, SLAF relies on building logical constraint formula based on a direct acyclic graph representation. FAMA takes as input partial GO execution traces, i.e. GO execution traces where some actions are missing, and observations are partial. This algorithm turns the task of learning into a planning problem, and it resolves it by using a classical planner. The AIA algorithm learns action models by using a query system. It generates plans and queries a black-box AI system with these plans to test them and updates its action model from the black-box responses. Then, The SRMLearn algorithm uses an alternative representation of input state-action relationships to learn an output action model. It represents a set of dependencies, intra-action and inter action, in the form of constraints of a weighted maximum satisfiability problem. These constraints are then solved with a weighted MAX-SAT solver. Then, the PDeepLearn algorithm is divided into three parts (1) PDeepLearn enumerates all possible candidate action patterns, (2) PDeepLearn identifies frequent action pairs and (3) PDeepLearn labels the action sequences in order to identify the ideal pattern. Then, the Louga [START_REF] Kucera | LOUGA: learning planning operators using genetic algorithms[END_REF] algorithm uses a genetic algorithm to learn the effects, and an ad-hoc algorithm to learn the preconditions. Finally, the LOCM family of action model learning approaches [START_REF] Cresswell | Acquiring planning domain models using LOCM[END_REF][START_REF] Cresswell | Generalised domain model acquisition from action traces[END_REF][START_REF] Gregory | Domain model acquisition in the presence of static relations in the LOP system[END_REF] works without information about initial, intermediate and final states. These algorithms extract, from GO execution traces, parameterized automata representing the behaviour of each object of the planning problems. Then, preconditions and effects are generated from these automata.

The second group of approaches takes as input random walks, i.e. sets of randomly generated action sequences. Random walk approaches like ERA (Balac et al., 2000a,b) and MARLIE [START_REF] Croonenborghs | Online learning and exploiting relational models in reinforcement learning[END_REF] deal with complete and noiseless observations and use reinforcement learning techniques. Also, approaches like IRALe (Rodrigues et al., 2010a) deal with complete but noisy observations. IRALe is based on an online active algorithm to explore and to learn incrementally the action model with noisy observations. Other approaches such as LSONIO [START_REF] Mourão | Learning STRIPS operators from noisy and incomplete observations[END_REF] deal with both partial and noisy observations. LSONIO uses a classifier based on a kernel trick method to learn action models. It consists of two steps: (1) it learns a state transition function as a set of classifiers, and (2) it derives the action model from the parameters of the classifiers.

Temporal Action Model Learning Some approaches have been proposed to learn temporal features [START_REF] Gabel | Online inference and enforcement of temporal properties[END_REF][START_REF] Neider | Learning linear temporal properties[END_REF][START_REF] Gaglione | Learning linear temporal properties from noisy data: A maxsat-based approach[END_REF][START_REF] Shah | Bayesian inference of temporal task specifications from demonstrations[END_REF], only [START_REF] Garrido | Learning temporal action models via constraint programming[END_REF] proposed an approach learning temporal action models.

HTN Task Model Learning: These approaches can be classified according to the output data of the learning process. The output can be the primitive task model, the set of HTN Methods and HTN Methods preconditions. Also, all these approaches take as input GO execution traces. A first group of approaches only learns the set of HTN Methods. First of all, Xiaoa et al. ( 2019) take as input a set of GO execution traces and HTN Methods and propose an algorithm to update incomplete HTN Methods by task insertions. HTN-Maker [START_REF] Hogg | Htn-maker: Learning htns with minimal additional knowledge engineering required[END_REF] and HTN-Maker ND [START_REF] Hogg | Learning hierarchical task networks for nondeterministic planning domains[END_REF] takes as input plan trace generated from STRIPS planner and annoted task provided by a planning expert. An annoted task is a triplet (n, Pre, E f f ) where n is a task, Pre is a set of propositions known as the preconditions and E f f is a set of atoms known as the effects. Then, [START_REF] Hogg | Learning methods to generate good plans: Integrating htn learning and reinforcement learning[END_REF] proposed an algorithm based on reinforcement learning. Then, [START_REF] Li | Learning probabilistic hierarchical task networks as probabilistic context-free grammars to capture user preferences[END_REF] proposed an algorithm taking as input only GO execution traces. This algorithm builds, from GO execution traces, a context free grammar (CFG) allowing to regenerate all plans. Then, methods are generated using CFG: one method for each production rule in the CFG. Then, [START_REF] Garland | Learning hierarchical task models by demonstration[END_REF] and [START_REF] Nargesian | Lhtndt: Learn htn method preconditions using decision tree[END_REF] proposed to learn HTN Methods from annotated plan. Annotated plans are plans segmented with the different tasks solved. However, obtaining these annotated examples is difficult and needs a lot of human effort.

A second group of approach learns HTN Methods preconditions. First of all, the CAMEL algorithm [START_REF] Ilghami | Camel: Learning method preconditions for HTN planning[END_REF] learns HTN Methods and the preconditions of HTN Methods from observations of GO execution traces, using the version space algorithm. This approach uses annotated task to build incrementally HTN Methods with preconditions. Then, the HDL algorithm [START_REF] Ilghami | Learning to do HTN planning[END_REF] takes as input GO execution traces. For each decomposition in GO execution traces, HDL checks if there exist a method responsible of this decomposition. If not, HDL adds a new method and initializes a new version space to capture its preconditions. Preconditions are learned in the same way as in the CAMEL algorithm.

Only HTN-Learner proposes to learn both action model and HTN Methods from decomposition trees. A decomposition tree is a tree corresponding to the decomposition of a method.

Although we focus on algorithms learning STRIPS, HTN and temporal action models, we can note that some approaches have also been proposed to learn ADL action models [START_REF] Shahaf | Learning partially observable action schemas[END_REF]Zhuo et al., 2010a), numerical features [START_REF] Martínez | Learning probabilistic action models from interpretation transitions[END_REF][START_REF] Gregory | Domain model acquisition in domains with action costs[END_REF][START_REF] Segura-Muros | Learning numerical action models from noisy and partially observable states by means of inductive rule learning techniques[END_REF] and multi-agent action models [START_REF] Zhuo | Learning action models for multi-agent planning[END_REF].

Conclusion

In this chapter, we introduced the AI planning concepts used in the development of this thesis and the literature related to these concepts. First of all, we presented the classical concepts of AI planning. Next, we presented three AI planning frameworks: (1) classical STRIPS planning and the PDDL language, (2) temporal planning and the PDDL 2.1 language and finally (3) hierarchical planning and the HDDL language. We have seen that these languages allow to declare the action models in the form planning domains. We then showed how to obtain these action models and we focused on ML-based approaches.

The ML-based approaches presented in this chapter have several drawbacks. The main drawback of these algorithms is that the learned action models are usually not correct enough to be usable by planners. The majority of the learning algorithms proposed so far do not provide any results on the ability of the learned action models to solve new problems, and a proofreading step of the learned action models is almost always required to correct errors. Moreover, most of the presented algorithms require GO execution traces, while we have seen that this kind of traces has the disadvantage to be biased. Finally, it is important that the learning algorithms are able to learn action models with partial and noisy observations. Indeed, in most cases these observations will be obtained using sensors, it is therefore possible that these different sensors cannot observe the environment in its entirety and it is also possible that there is noise in the observations due to the imprecision of these sensors.

As we have mentioned in this chapter, a planning problem can be seen as the search for a path in a state machine. Moreover, state machines are equivalent to automata. As we will see in the next chapter, an automaton allows to represent a grammar. We will see in the second part of this thesis that STRIPS planning problems can be represented using regular grammars. It is therefore possible to learn a regular grammar representing the state machine related to a planning problem. This is the basis of our approach: learning the state machine related to the planning problem using Regular Grammar Induction algorithms (see Chapter 3) and inducing the action model from this grammar. Finally, as planning problems are declared using a planning domain, our approach will have to represent the action model in the form of a planning domain, and more precisely, in the form of a set of planning operators. We will also see that it is possible to extend this approach for planning problems relaxing some assumptions made by STRIPS planning.

Before presenting our contributions in more details, we will give in the next chapter an introduction to the regular grammar induction field.

Chapter 3

Regular Grammar Induction

Introduction

We are interested in learning structural patterns that can be described by a grammar. A grammar describes how to form strings, i.e. sequences of symbols, belonging to a language. This problem belongs to the grammatical induction field. Grammatical induction is the process of discovering an acceptable grammar for a language from a training dataset. This dataset consists of at least one positive sample, i.e. a finite subset of a language. Grammatical induction is therefore the process of discovering a grammar from a language containing at least all the elements of the positive sample. We can also have a negative sample, i.e. a finite set of sequences not belonging to the language. In this case, grammatical induction is the process of discovering a grammar from a language containing at least all the positive sample and containing no element of the negative sample. Grammatical induction has been studied since the development of the theory of formal grammars. Besides its theoretical interest, it offers a set of potential applications, in particular in the fields of Syntactic and Structural Pattern Recognition, Natural Language Processing [START_REF] Adriaans | Computational grammar induction for linguists[END_REF][START_REF] Dupont | The QSM algorithm and its application to software behavior model induction[END_REF][START_REF] Boström | Theory-guided induction of logic programs by inference of regular languages[END_REF][START_REF] Boström | Predicate invention and learning from positive examples only[END_REF][START_REF] Cruz-Alcázar | Learning regular grammars to model musical style: Comparing different coding schemes[END_REF][START_REF] Bex | Inference of concise dtds from XML data[END_REF][START_REF] Cruz-Alcázar | Two grammatical inference applications in music processing[END_REF][START_REF] Stein | The iupac chemical identifier -technical manual[END_REF][START_REF] Bréhélin | Hidden markov models with patterns to learn boolean vector sequences and application to the built-in selftest for integrated circuits[END_REF][START_REF] Raffelt | Learnlib: A library for automata learning and experimentation[END_REF][START_REF] Berg | Regular inference for state machines with parameters[END_REF]. We will limit ourselves to regular induction, i.e. the learning of a grammar representing a regular language. Indeed, as we will see in the next part, planning problems can be represented by regular grammars.

In this chapter we present the different notions of automata theory useful for grammatical induction. Once these notions are presented we give the formal framework of grammatical induction. More precisely, we will present the notion of identification in the limit and the search space explored by the regular grammar induction algorithms. Finally we present several approaches to grammatical induction. And more precisely, we are interested in regular grammar induction algorithms using positive and negative examples, because they allow us to learn regular languages in the limit.

Please note that for the sake of coherence and consistency with the rest of this thesis, we will not use traditional notations.

Definitions and Notations

We start by introducing the basic notions of automata theory. These notions will be useful to give the formal framework of grammatical induction.

Basic Notions of Automata Theory

We denote A a finite non-empty alphabet, i.e. the set of symbols present in the language. Then, we denote u, v, w the elements of A * , i.e. the sequences of finite length on A. Finally, we denote |u| the length of the sequence u and the empty sequence. Definition 3.1 u is a prefix of v if there exists w such that uw = v. Definition 3.2 u is a suffix of v if there exists w such that wu = v. Example 3.1 Let's take the alphabet A = {a, b}, with the sequence v = aababbba. The sequence u = aaba is a prefix of v and w = bbba is a suffix of v. Definition 3.3 A language L is a subset of A * . Example 3.2 Let L = (a * ba * b) * be the language accepting an even number of b, all sequences u such that u contains an even number of b belong to the language L (denoted u ∈ L). Every sequence u such that u contains an odd number of b do not belong to the language L (denoted u ∈ L). In our example, we have abbbab ∈ L and abbab ∈ L.

In the rest of this chapter, we denote L even the language accepting an even number of b. Definition 3.4 Let Pr(L) = {u|∃v, uv ∈ L} be the prefix set of L and L/u = {v|uv ∈ L} be the right quotient of L. We have L/u = ∅ if and only if u ∈ Pr(L).

Example 3.3

Let's continue with our example, we have v = abbaaabba ∈ L even , u = ab is a prefix of v and w = ba is a suffix of v. Also, we have ab ∈ Pr(L even ) and the right quotient is:

L even /ab = a * b(a * ba * b) * .

Regular Grammar

A regular grammar (RG) is a grammar describing regular language [START_REF] Yu | Regular languages[END_REF]). An RG can be right or left:

• A right RG has the form:

• S → aS • S → a • S → a • S → aS • A left RG has the form: • S → S a • S → a • S → a • S → Sa
where S and S are non-terminal symbols, and a is a terminal symbol. A terminal symbol is a symbol belonging to the alphabet A and a non-terminal symbol is a which can be replaced according to the above rules. The notion of DFA will be discussed in Section 3.2.3.

Example 3.4

Let's take as example the following right RG:

• S 0 → • S 0 → a • S 0 → aS 0 • S 0 → bS 1 • S 1 → bS 0 • S 1 → b • S 1 → aS 1
This right RG accepts the even language L even and can be represented by the DFA in Figure 3.1.

In the rest of this chapter, we denote Σ even this DFA accepting the even language L even .

Automata and Partitions

Automata

Definition 3.5 An automaton is a tuple Σ = (S, A, γ, s 0 , G) where S is a finite state set, A is a finite alphabet, γ is a transition function such that S × A → S, s 0 ∈ S is the initial state and G ⊆ S * is the goal state set.

Informally, an automaton is a set of states, linked together by transitions labeled by symbols. Given a sequence as input, the automaton reads the symbols of the sequence one by one and goes from state to state according to the transitions. The sequence read is either accepted by the automaton or rejected. A sequence is accepted if the automaton can read all the symbols of the sequence, and if the last state reached is a goal state. Definition 3.6 A DFA is a finite automaton whose transitions are deterministic, i.e. for each state, there is a unique transition for a read symbol. Definition 3.7 Let γ : S × A * → S be the transition function extended to sequences. This function returns the last reached state by the automaton after reading the sequence from a given state. This function is defined as follows:

∀s ∈ S, ∀u ∈ A * , γ(s, u) = s if u = . γ(γ(s, u ), a) if u = u a.
We can formally define the acceptation of a sequence u by an automaton Σ: it accepts a sequence u if and only if γ(s 0 , u) ∈ G. Finally, the set of sequences accepted by an automaton Σ is denoted L(Σ) and the language accepted by Σ is defined as follows:

L(Σ) = {u ∈ A * s.t. γ(s 0 , u) ∈ G}.
Example 3.5 Let's take the automaton in Figure 3.1. In this example, the automaton states are S = {0, 1}. Then, the alphabet is A = {a, b}. Then, the transition function is represented by arcs connecting two states. For example, we have γ(0, b) = 1. Then, the goal state is: G = {0} and the initial state is s 0 = 0. This automaton accepts the sequence abba because γ(0, abba) = 0 ∈ G and rejects, i.e. does not accept, the sequence abbaba because γ(0, abbaba) = 1 ∈ G. Finally, we have abba ∈ L(Σ), abbaba ∈ L(Σ).

A same language can be accepted by several automata. Nevertheless, in the field of grammatical induction, there is a form of automaton of particular interest to us: the minimal canonical automaton A(L)1 . Informally, A(L) is the smallest automaton accepting the language L. For example, let's take the even language L even . The automaton in Figure 3.2a gives its minimal canonical automaton. And Figure 3.2b shows another automaton, non canonical, accepting this language. We note that if we merge states 1 and 2 of the automaton in Figure 3.2b we obtain the automaton in Figure 3.2a. This merge is a derivation of the automaton3.2b.

Quotient Automata

Let's go back to the automata in Figures 3.2a and 3.2b. Let's assume that we have S = {0, 1, 2}, we denote Π i = {{0}, {1, 2}} and Π j = {{0}, {1}, {2}} two sets of subsets of S. We call these sets partitions. Π i is the partition of S for the automaton in Figure 3.2a and Π j is the partition of S for the automaton in Figure 3.2b. Both automata share the same state set S but Figure 3.2a considers that states 1 and 2 are the same state, while Figure 3.2b considers them as distinct states. The subsets of these partitions are called blocks. For Π i blocks are: {0} and {1, 2}. And for Π j blocks are: {0}, {1} and {2}. We can note that Π i is smaller than Π j . Indeed, Π i contains two blocks while Π j contains three blocks. This is due to Π i being finer than Π j . A partition Π i is finer than a partition Π j when all blocks of Π i are either a block or the union of several blocks of Π j . Finally, we can note that, when we merge the blocks {1} and {2} of Π j , we obtain the automaton 3.2a. The automaton 3.2a is the quotient automaton of 3.2b w.r.t. the partition Π i .

We will now formally define these different notions.

Definition 3.8 For any set S, a partition Π is a set of subsets of S, non-empty and disjoint two by two whose union is S. If s ∈ S then B(s, Π) is the single block of Π containing s. Definition 3.9 A partition Π i is finer than a partition Π j if each block of Π j is either a block of Π i or the union of several blocks of Π j .

Definition 3.10 Let Σ = (S, A, γ, s 0 , G) be an automaton. The automaton Σ/Π = (S , A, γ , B(s 0 , Π), G ) is derived from Σ w.r.t. the partition Π, and is called the quotient automaton Σ/Π. This automaton is defined as follows:

S = S/Π = {B(s, Π)|s ∈ S}, G = G/Π = {B(g, Π)|g ∈ G}, γ = S × A → 2 S : ∀s i , s j ∈ S , ∀a ∈ A, s j ∈ γ (s i , a) iff ∃s i , s j ∈ S s.t. s i ∈ B(s i , Π) ∧ s j ∈ B(s j , Π) ∧ s j ∈ γ(s i , a).

Samples of a Language and Associated Automata

Before giving the formal framework of regular grammatical induction (RGI), we will define what a sample is and give some remarkable automata used in the RGI field. Definition 3.12 A negative sample, denoted I -, is a finite subset of sequences of the language L, the complement of L: L = A * \ L.

Example 3.6 Let's go back to the even language L even . The complement of this language is L = {a * ba * {a * ba * ba * } * }; the language accepting an odd number of b. A positive sample of L even could be I + = {a, abb, abbabb, . . .} and a negative sample could be I -= {b, babb, bbb, . . .}.

Definition 3.13

A positive sample I + is structurally complete [START_REF] Fu | Grammatical inference: Introduction and survey -part II[END_REF] to the automaton Σ accepting the language L if there exists an acceptation of I + such that all automaton's transitions are used and all goal states of G are used as goal states.

Example 3.7 Let's go back to the automaton Σ even (see Figure 3.1) accepting the even language L even . The positive sample I + = { , a, abb, abab} is structurally complete but the positive sample I + = { , a, abb} is not structurally complete because the transition γ(1, a) is not used.

Now that we have given the definitions related to sampling, we will give some remarkable automata used for regular grammar induction. Definition 3.14 The maximal canonical automaton w.r.t I + , denoted MCA(I + ), is defined as follows:

S = {S i,j |1 ≤ i ≤ |I + |, 1 ≤ j ≤ |v i |, v i,j = a i,1 , a i,2 , . . . , a i,j } ∪ { }, s 0 = , G = I + , γ , a = {a i,1 |a i,1 = a, 1 ≤ i ≤ |I + |}, γ(v i,j , a) = {v i,j+1 |v i,j+1 = v i,j a, 1 ≤ i ≤ |I + |, 1 ≤ j ≤ |v i | -1}.
Example 3.8 Suppose we have I + = {a, abb, bab}. The alphabet is A = {a, b}. The state set is: 2,3,4,[START_REF] Grand | Action Model Learning based on Grammar Induction[END_REF]6, 7}.

S = {S i,j |1 ≤ i ≤ |I + |, 1 ≤ j ≤ |v i |, v i,j = a i,1 , a i,2 , . . . , a i,j } ∪ { }, = {S i,j |1 ≤ i ≤ |{a, abb, bab}|, 1 ≤ j ≤ |v i |, v i,j = a i,1 , a i,2 , . . . , a i,j } ∪ {0}, = {1, 2, 3} ∪ {4, 5} ∪ {6, 7} ∪ {0} = {0, 1,
Then, s 0 = 0. Then, goal states are states defined by the last symbol of each example then G = {1, 6, 7}. Then, we build transition starting from the initial state s 0 : γ(0, a) = ({a, abb, bab}). Therefore, L(MCA(I + )) = I + and MCA(I + ) is the bigger automaton (possibly non-deterministic), i.e. the automaton with the maximal number of states, for which I + is structurally complete. We can note that MCA(I + ) is possibly non-deterministic.

Definition 3.15

The prefix tree acceptor of I + , denoted PTA(I + ) is the automaton MCA(I + )/Π where the partition Π is defined as follow:

∀s, s ∈ S, B(s, Π) = B(s , Π) i f f Pr(s) = Pr(s ).
Less formally, the PTA(I + ) is the determinization of MCA(I + ). Then, to build PTA(I + ) we merge all states sharing the same prefix. Therefore, L(PTA(I + )) = I + and PTA(I + ) is the biggest DFA, i.e. the DFA with the maximal number of states, for which I + is structurally complete.

Example 3.9 Suppose we have I + = {a, bab, abb}. The set state is S = {0, 1,2,3,4,[START_REF] Grand | Action Model Learning based on Grammar Induction[END_REF]6, 7} and the partition after merging is Π = {{0}, {1, 2}, {3}, {4}, {5}, {6}, {7}}. Definition 3.17 The automaton A(L) is the smallest automaton for the language L. A(L) is called the minimal canonical automaton of L and is defined as follows:

S = {L/u|u ∈ Pr(L)}, s 0 = L/ , G = {L/u|u ∈ L}, γ(L/u, a) = L/ua, ua ∈ Pr(L).
Example 3.11 Let's take the the even language L even . The alphabet is A = {a, b}, the state set is build using prefixes : S = {0, 1}, we have s 0 = 0 and G = {0}. Finally, we build the transition function: γ(0, a) = 0, γ(0, b) = 1 etc. Figure 3.6 gives Σ even , the minimal canonical automaton accepting L even .

Regular Grammatical Induction

Now that we have seen the different notions around the theory of automata used for regular grammatical induction (RGI), we will give the formal framework of RGI. First of all, we will describe the notion of identification in the limit [START_REF] Gold | Language identification in the limit[END_REF]. An RGI algorithm identifies a language L in the limit, if, from a finite set of training data, this algorithm identifies L, and, if we add more data, continues to identify L. We will show in this section that all regular languages are identifiable in the limit when the training datasets contain both positive and negative examples. Then, we will introduce the search space used by RGI algorithms dealing with both positive and negative examples.

Identification in the limit

We will introduce the notion of identification in the limit.

Suppose we have an infinite ordered set of positive examples, including any element of our language L, and an infinite set of negative examples including any element of the complement language. And let us denote ω i the i th training data, with ω i either a positive or a negative example. Let us also assume that we have H the hypotheses space, i.e. the set of hypotheses. An hypothesis is a possible concept or model for the training datasets. For example, in the context of grammar induction, the set of hypotheses could the set of automata accepting the positive sample. Also, we denote H(ω i ) the set of correct hypotheses after reading the first i training data, i.e. the set of hypotheses covering all the first i training data.

Let's step out of the RGI framework for a moment. Suppose we want to identify a set of numbers with the following training dataset: Ω = {{0, 1}, {1, 1}, {2, 1}, {3, 1}, {4, 0}, {5, 1}, {6, 0}, {7, 0}, {8, 1} . . .} with H = {N, even, odd, Fibonacci} our hypotheses space.

• ω 0 = 0 is a positive example: Several hypotheses can identify a set beginning by 0, the set of natural integers N, the Fibonacci sequence, the set of even integers. Only the odd hypothesis is incorrect.

• ω 1 = 1 is a positive example: We have H(ω 1 ) = {N, Fibonacci}. Both even and odd hypotheses are incorrect.

• ω 2 = 2 is a positive example: We have H(ω 2 ) = {N, Fibonacci}. Both even and odd hypotheses are incorrect.

• ω 3 = 3 is a positive example: We have H(ω 3 ) = {N, Fibonacci}. Both even and odd hypotheses are incorrect.

• ω 4 = 4 is a negative example: We have H(ω 4 ) = {Fibonacci}. The Fibonacci sequence in the only correct hypothesis.

• ω 5 = 5 is a positive example: We have H(ω 5 ) = {Fibonacci}. The Fibonacci sequence in the only correct hypothesis.

• ω 6 = 6 is a negative example: We have H(ω 6 ) = {Fibonacci}. The Fibonacci sequence in the only correct hypothesis.

• ω 7 = 7 is a negative example: We have H(ω 7 ) = {Fibonacci}. The Fibonacci sequence in the only correct hypothesis.

• ω 8 = 8 is a positive example: We have H(ω 8 ) = {Fibonacci}. The Fibonacci sequence in the only correct hypothesis.

We notice that from ω 4 only the Fibonacci hypothesis is correct, and this hypothesis remains correct after adding all positive and negative examples. This is the identification in the limit. Now that we have seen an introductory example about the notion of identification in the limit, we will formally introduce this notion. Definition 3.18 A positive presentation of a language L is an infinite set of examples with at least one occurrence of each element of L. These elements are the positive examples of the language L. Definition 3.19 A negative presentation of a language L is an infinite set of examples with at least one occurrence of each element of the complement language L. These elements are the negative examples (or counter examples) of the language L. Definition 3.20 A complete presentation of a language L is an infinite and ordered set

Ω = {(ω i , d i ) : A * × {0, 1}} with: d = 1 If ω i ∈ L. 0 Otherwise.
A positive sample is a finite set of elements of a language. We can therefore consider a positive presentation as an infinite sequence of positive samples whose size increases. Such a presentation is called admissible. Notions of identification in the limit and class of languages have been introduced by [START_REF] Gold | Language identification in the limit[END_REF] and are defined as follows: Definition 3.21 Let M be an inference method and Ω an admissible presentation of a language L. We denote ω i the i th element of Ω. The inference method M has to propose a new solution in the search space for each new example. Let H(ω i ) be the hypotheses proposed by M after reading the i first elements of Ω. A method M identifies in the limit the language L if and only if there exists a finite index j such that:

1. ∀i ≥ j, H(ω i ) = H(ω j ), 2. H(ω j ) = {L}.
The first condition ensures that M converges and the second condition ensures that M converges to a correct representation of L. Definition 3.22 An inference method M identifies a class C of languages in the limit if M identifies in the limit all languages L ∈ C. Theorem 3.1 The class of regular languages is identifiable in the limit in a polynomial time for an inference method M using an admissible presentation containing both positive and negative samples.

In this chapter we focus on the notion of identification in the limit because it is an exact identification criterion. Nevertheless, many other identification criteria have been proposed. For example, the BC-identification (Behaviorally Correct Identification) [START_REF] Barzdinš | Two theorems on the limiting synthesis of functions[END_REF][START_REF] Case | Comparison of identification criteria for machine inductive inference[END_REF] does not require that the inference method converge to a single hypothesis in the space of hypotheses, but that, from the point of convergence, all the hypotheses proposed constitute a possible description of the correct solution.

Search Space

We will now focus on the search space of RGI algorithms using both positive and negative samples. First we will present, in a general framework, the notion of lattice. Then, we will show how to use the lattice in the RGI field.

A lattice is a partially ordered set with a greatest upper and a smallest lower bound. Each pair of elements of a lattice has a smallest lower bound and a greatest upper bound. We can formally define a lattice as follows: Definition 3.23 A lattice is a pair (S, ≤) such that:

• ≤ is an ordering relation of S such that:

• ∀x ∈ S, x ≤ x, • ∀x, y ∈ S, x ≤ y ∧ y ≤ x =⇒ x = y, • ∀x, y, z ∈ S, x ≤ y ∧ y ≤ z =⇒ x ≤ z.
• All pairs {x, y} of S has a greatest upper and a smallest lower bound:

• the smallest lower bound of x and y, denoted x y is the unique maximal element of the set of predecessors of x and y,

• the greatest upper bound of x and y, denoted x⊥y is the unique maximal element of the set of successors of x and y.

Property 3.2 If a partition Π i is finer than a partition Π j , denoted Π i ≤ Π j , then quotient automata follow the same relation: Σ/Π i ≤ Σ/Π j , and L(Σ/Π i ) ⊆ L(Σ/Π j ).

Definition 3.24

The set of quotient automata of Σ with the partial order ≤ is an automata lattice, denoted Lat(Σ), for whom Σ and U A are, respectively, the smallest lower and the greater upper bound.

Theorem 3.2 Let I + be a positive sample of the language L and Σ an automaton such that L(Σ) = L. If I + is structurally complete w.r.t Σ, then Σ ∈ Lat(MCA(I + )).

Theorem 3.3 Let I + be a positive sample of the language L and C(L) the canonical automaton of L. If I + is structurally complete w.r.t A(L), then we have A(L) ∈ Lat(PTA(I + )).

We can consider the grammatical induction by the search of an automaton compatible with I + and I -. Formally, we search an automaton Σ such that (1) I + ⊆ L(Σ) and ( 2) I -∩ L(Σ) = ∅. There exist several automata checking both conditions. For example, MCA(I + ) checks both conditions. However, this automaton does not generalize the positive sample. So, we have to search the most general solution w.r.t the negative sample I -. If we choose the simplicity of the inferred automaton as a generality criterion and restrict the search to DFA, then the solution we are looking for is the compatible DFA with the minimum number of states. This is the so-called minimal DFA consistency problem which is NP-hard [START_REF] Angluin | On the complexity of minimum inference of regular sets[END_REF][START_REF] Gold | Complexity of automaton identification from given data[END_REF]. We are therefore looking for an element of the border set of our automata lattice. The border set [START_REF] Dupont | What is the search space of the regular inference?[END_REF] is the limit of the possible generalization from a positive sample and a negative sample.

Example 3.12 Suppose we have I + = {a, bab} and I -= {ababb, b, baa}. Figure 3.7 shows the automata lattice whose smallest lower bound is MCA(I + ) and greatest upper bound is U A. Automata in the green area are automata compatible with I -and automata in the red area are automata incompatible with I -. The border set, denoted BS MCA (I + , I -), is the border between the green area and the red area.

Regular Grammar Induction Algorithms

We focus on RGI algorithms taking as input both positive and negative samples. Unlike algorithms using only a positive sample, algorithms that take both positive and negative samples as input can identify the class of regular languages in the limit. Also, this identification is done in a polynomial time with samples of polynomial size w.r.t the number of states. These properties are particularly interesting in applications where the dataset acquisition is a difficult and costly task. Let us note however that there are several inference algorithms using only positive samples e.g. ID [START_REF] Angluin | A note on the number of queries needed to identify regular languages[END_REF], IID [START_REF] Parekh | A polynomial time incremental algorithm for learning dfa[END_REF], k-TSSI (Garcia and Vidal, 1990), MGGI (Garcia et al., 1990), ECGI [START_REF] Kudo | Efficient regular grammatical inference techniques by the use of partial similarities and their logical relationships[END_REF] etc. A first method of inference is the use of genetic algorithm [START_REF] Goldberg | Genetic Algorithms in Search Optimization and Machine Learning[END_REF][START_REF] Mitchell | An introduction to genetic algorithms[END_REF]. For example, the GIG algorithm evolves in the lattice Lat(MCA(I + )) a population obtained by deriving MCA(I + ) using structural mutation and crossover.

Then, another approach is to build the border set and returns the smallest automaton belonging to the border set. For example, the RIG algorithm (Regular Inference of Grammar) [START_REF] Miclet | Inférence grammaticale à partir d'exemples et de contre-exemples: deux algorithmes optimaux:(big et rig) et une version heuristique (brig)[END_REF] builds the border set BS MCA (I + , I -). It proceeds by enumerating the automata derived from MCA(I + ). This is a breadth-first search of Lat(MCA(I + )) keeping only compatible automata at each depth in the lattice. At each step, the algorithm performs a hierarchical pruning, i.e. it eliminates all automata at depth i+1 that derive from at least one incompatible automaton. Finally, the border set is obtained by storing all the compatible automata that have no compatible derivatives.

The last approach is to test block merging until there are no more possible merge. The EDSM algorithm (Evidence Driven State Merging) [START_REF] Lang | Evidence driven state merging with search[END_REF] begins by building the prefix tree acceptor PTA(I + ). Then, the algorithm tests all block mergings compatible with I -. To find blocks to merge, EDSM uses a heuristic that computes a score for each state of the automaton. Then, it tests the pairs of blocks with the highest score. Then, the RPNI algorithm (Regular Positive and Negative Inference) [START_REF] Oncina | Inferring regular languages in polynomial updated time[END_REF]

3 for i ← 1 : |Π| -1 do 4 for j ← 0 : i -1 do 5 Π ← Π/{B j , B i } ∪ {B i ∪ B j }; 6 Σ/Π ← derive(Σ/Π ); 7 Σ/Π ← deterministic_merge(Σ/Π ); 8 if compatible(Σ/Π , I -) then 9 Σ ← Σ/Π ; 10 Π ← Π ; 11 break; 12 return Σ, Π
search in Lat(PTA(I + )) and finds a locally optimal solution to the minimal DFA consistency problem. This algorithm relies on the structur completeness of the positive sample I + . RPNI starts by creating a partition of the set of states of PTA(I + ). RPNI then proceeds to merge the blocks that do not cause compatibility problems with I -according to the partial order induced by the lattice.

Focus on the RPNI Algorithm

Among all RGI algorithms, RPNI is very efficient and has all the right properties: (1) it is able to identify in the limit the class of the regular languages; (2) RPNI is locally optimal.

The RPNI Algorithm

The RPNI algorithm is described by Algorithm 1 and retains only merges which do not cause the acceptation of a negative example (line [3][4][START_REF] Grand | Action Model Learning based on Grammar Induction[END_REF][6][7][8][9][10][11]. RPNI begins by testing the merge between states 0 and 1. RPNI updates the partition by merging blocks where states 0 and 1 appear (line 5): Π = {{0, 1}, {2}, {3}, . . . , {10}}. Then, RPNI computes the quotient automaton Σ /Π (line 6). As the quotient automaton is deterministic and rejects all negative examples, the merge is retained (line 8-10). Now, RPNI tries to merge states 0 and 2. We can observe that the automaton is non-deterministic. Indeed, state 0 has now two outgoing transitions a. To remove the non-deterministic transitions, RPNI performs a deterministic merge (line 7). Figure 3.9 shows an execution of the deterministic merge. RPNI begins by merging states 0 and 3. As the automaton is always nondeterministic (the automaton has two outgoing transitions a and b in the state 0), RPNI merges states 0 and 4, and states 0 and 5. The automaton obtained after these mergings is now deterministic. However, it accepts all negative examples, then RPNI goes back to the previous successful merge and tries to merge the next pair of sates 0 and 3. As automaton is non-deterministic, RPNI performs the deterministic merge.

After determinization, the automaton accepts the negative example ba, the merge is therefore not retained. After having testes all state merges, the final partition is Π = {{0, 1, 4, 6, 8, 9, 10}, {2, 3, 5, 7}}.

We can now study the complexity of the RPNI algorithm. We have |I + | the number of states in PTA(I + ), and we denote |I -| = ∑

C RPN I (I + , I -) = O(|I + | 2 .(|I + | + |I -|))

Identification in the Limit and Characteristic Sample Specification

The RPNI algorithm identifies in the limit the class of regular languages. In other words, when the sample is characteristic of a language L, then RPNI learns the minimal canonical automaton of A(L). We will define notion of characteristic sample in this section. The notion of characteristic sample Ω c = (I c + , I c -) was introduced by [START_REF] Dupont | Incremental regular inference[END_REF].

The RPNI algorithm identifies the class of regular languages in the limit. Then, for each regular language L there exists a sample Ω c = {I c + , I c -} such that:

• RPN I(I c + , I c -) → Σ with Σ = A(L).
• ∀I + , I -such that ∀ω ∈ I + , ω ∈ L and ∀ω ∈ I -, ω ∈ L(P) then RPN I(I c + ,

I c -) = RPN I({I c + ∪ I + }, {I c -∪ I -}).
The sample Ω c is the characteristic sample of L(P). We give the conditions that the samples must satisfy to be characteristic for the language L and the RPNI algorithm. To this end, we shall present the notions of the set of short prefixes and the kernel of a language [START_REF] Oncina | Inferring regular languages in polynomial updated time[END_REF].

Definition 3.25

The set of short prefixes S P (L) for a language L is defined as follows:

S P (L) = {x ∈ Pr(L)|¬∃u ∈ A * s.t. L/u = L/x ∧ u < x}.
Less formally, the set of short prefixes S P (L) is the set of prefixes reaching to a state of the canonical automaton of L.

Definition 3.26

The kernel N(L) for a language L is defined as follows:

N(L) = { } ∪ {xa|x ∈ S P (L) ∧ a ∈ A ∧ xa ∈ Pr(L)}.
Example 3.14 For example, consider the DFA Σ even , the minimal canonical automaton of the even language L even . For this DFA S P (L) = { , b}. And the kernel of a language is the set of elements of S P (L) to which each transition of the minimal canonical automaton has been added. Let us take again the DFA Σ even , for this DFA we have N(L) = { , a, b, ba, bb}.

Finally, we can build the characteristic sample for the language L: Definition 3.27 The sample Ω c = (I c + , I c -) is characteristic for the language L if and only if:

1. ∀x ∈ N(L) if x ∈ L then x ∈ I c + ∨ ∃u ∈ A * |xu ∈ I c + , 2. ∀x ∈ S P (L), ∀y ∈ N(L) if L/x = L/y then ∃u ∈ A * |(xu ∈ I c + ∧ yu ∈ I c -) ∨ (xu ∈ I c -∧ yu ∈ I c + ).
The first condition requires that all the kernel elements are present in our positive sample. When an element of the kernel belongs to L, then it belongs to our positive sample, but when it does not belong to L then this element is the prefix of an element of the positive sample. And the second condition requires that for each pair of elements of S P (L) and N(L) leading to distinct states, there exists a suffix u which distinguishes the two states reached in our sampling. 3.1: Characteristic sample specification for the even language Example 3.15 Let's take again the DFA Σ even , we have:

S P (L) = { , b}, N(L) = { , a, b, ba, bb}.
To build a characteristic sample for this DFA we have to build a sample respecting the two conditions seen previously. The first condition requires that all the elements of the kernel are either positive examples or are prefixes of positive examples. We start by recovering the elements of the kernel which are sequences accepted by the automaton. So we have: N(L) ∩ L = { , a, bb}. Now we have to find positive examples whose suffixes would be b and ba. We can use bb whose prefix is b, and baaaba whose prefix is ba. So the first condition gives us the following positive sample: I c + = { , a, bb, baaaba}. We must now build the negative sample, and complete the positive sample to satisfy the second condition. The second condition requires that for each pair of elements of N(L) and S P (L) leading to two distinct states, there is a positive example, and a negative example distinguishing them. For example, if we take the pair ( , a), both sequences reach the same state. On the other hand if we take ( , b), reaches the state 0, while b reaches the state 1, so we have to add a positive and a negative example for this pair. We can use u = a, which implies that we have a ∈ I c + and ba ∈ I c -. Table 3.1 gives in detail the whole specification of the positive and negative samples. Finally, the characteristic sample for the even language is:

I c + = { , a, baaaba, bab, bb, bba}. I c -= {ab, b, ba}.
Please note that a characteristic sample is not unique, indeed, several suffix u can be used to satisfy the first or the second condition. Finally, we can now compute the size of the characteristic sample:

|S P (L)| = |S|. |N(L)| = O(1 + |S| + |A|). |I c + | = O(|S| 2 .|A|). |I c -| = O(|S| 2 .|A|).

Conclusion

To conclude, we have seen in this chapter the theoretical framework of RGI as well as a state of the art on RGI using both positive and negative samples. Note that grammatical induction techniques have been proposed to learn other forms of grammars such as transducers [START_REF] Oncina | Learning subsequential transducers for pattern recognition interpretation tasks[END_REF][START_REF] Vilar | Improve the learning of subsequential transducers by using alignments and dictionaries[END_REF], probabilistic grammars [START_REF] Angluin | Queries and concept learning[END_REF][START_REF] Stolcke | Inducing probabilistic grammars by bayesian model merging[END_REF][START_REF] Carrasco | Learning stochastic regular grammars by means of a state merging method[END_REF][START_REF] Guttman | Probabilistic automata and distributions over sequences[END_REF], context-free grammars [START_REF] Sakakibara | Efficient learning of context-free grammars from positive structural examples[END_REF][START_REF] Nevill-Manning | Identifying hierarchical structure in sequences: A linear-time algorithm[END_REF][START_REF] Koshiba | Inferring pure context-free languages from positive data[END_REF][START_REF] Eyraud | Lars: A learning algorithm for rewriting systems[END_REF] etc. We focused on RGI techniques because, as we will see later, planning problems are related to state machines that represent regular languages.

We have focused on algorithms using both positive and negative samples because these algorithms can identify in the limit the class of regular languages in a polynomial time. In addition, we specified the sampling characterization for the RPNI algorithm. Although there are other identification criteria, identification in the limit is an exact identification criterion. This is of interest to us because we will use RGI techniques to learn action models. As we will see next, we will use RGI techniques, and more precisely the RPNI algorithm, to learn a language representing a planning problem and then we will induce the action model from the language. Using RGI techniques with an exact identification criterion will allow us to avoid learning errors in the learned action model.

59

Chapter 4

STRIPS Action Model Learning

Introduction

As mentioned in Chapter 2, a planning problem can be seen as the search for a path in a state machine. Moreover, state machines are equivalent to automata. As we have seen in Chapter 3, an automaton allows to represent a grammar. As we will see in this chapter, these grammars are regular. It is therefore possible to identify in the limit the regular grammar related to a given planning problem using algorithm taking as input both positive and negative samples such as RPNI [START_REF] Oncina | Inferring regular languages in polynomial updated time[END_REF]. In this chapter, we will exploit this property to learn action models.

As mentioned in Chapter 2, hand-encoding action models is a difficult task which makes it difficult to use AI planning techniques in real-world applications. To mitigate this difficulty, the AI planning community have proposed approaches using machine learning techniques to compute action models from execution traces such as, for example, ARMS [START_REF] Yang | Learning action models from plan examples using weighted MAX-SAT[END_REF], SLAF [START_REF] Shahaf | Learning partially observable action schemas[END_REF], LSONIO [START_REF] Mourão | Learning STRIPS operators from noisy and incomplete observations[END_REF], LOCM [START_REF] Cresswell | Acquiring planning domain models using LOCM[END_REF]. These approaches have three main drawbacks:

1. Most of these approaches require a lot of data to perform the learning of action models and in many real world applications, acquiring training datasets is difficult and costly, e.g., Mars Exploration Rover operations [START_REF] Bresina | Activity planning for the mars exploration rovers[END_REF] or fleet of Autonomous Underwater Vehicles for offshore missions [START_REF] Carreno | Task allocation strategy for heterogeneous robot teams in offshore missions[END_REF][START_REF] Lesire | A distributed architecture for supervision of autonomous multi-robot missions[END_REF]).

2. The learned domains are not enough accurate to be used "as is" in a planner: a step of expert proofreading is still necessary to correct them. Even small syntactical errors can make sometime the learned domains useless for planning.

3. Even if some approaches, e.g., [START_REF] Mourão | Learning STRIPS operators from noisy and incomplete observations[END_REF][START_REF] Segura-Muros | Learning numerical action models from noisy and partially observable states by means of inductive rule learning techniques[END_REF]Rodrigues et al., 2010a) are able to learn from noisy and/or partially observable data, few approaches are able to handle very high levels of noise and high levels of partial observations as can be encountered in real world applications.

In this chapter, we present AMLSI (Action Model Learning with State machine Interaction) (Grand et al., 2020b,a), a novel approach learning STRIPS action models based on grammar induction techniques (see Chapter 3). The idea is to learn the state machine related to the planning problem using RGI algorithms and inducing the action model from this state machine. Also, as planning problems are declared using a planning domain, our approach will have to represent the action model in the form of a planning domain.

The key idea of the AMLSI approach is to interact with the environment in which the agent will have to solve planning problems to learn the action model. AMLSI tests different randomly choosen actions, observes how the environment evolves when these actions are executed, learns the DFA and induces the action model from its observations. AMLSI does not require any prior knowledge regarding the feasibility of actions in a given state, and state observations can be partial and noisy. AMLSI is highly accurate even with highly partial and noisy state observations. Our contribution is threefold: 2. AMLSI is highly accurate and requires few data.

3. AMLSI is robust to partial and noisy observations.

In the rest of this chapter, we start by giving the problem statement of the STRIPS Action Model Learning. More precisely, we extend the formal framework proposed in Chapter 2 to the learning problem. Then, we detail the AMLSI approach. We present the different steps of our approach, and we give the formal properties of our approach. Finally, we propose an evaluation of our approach. We compare AMLSI with state-of-the-art approaches and show that AMLSI outperforms these approaches.

Problem statement

Without loss of generality, we propose a formal framework inspired by [START_REF] Höller | Assessing the expressivity of planning formalisms through the comparison to formal languages[END_REF] in order to define the STRIPS learning problem as the lifting of a state transition system into a propositional language. This formal framework extends the formal framework proposed in Chapter 2 to the learning problem. Definition 4.1 A classical planning problem P extended to the learning problem is a tuple (L, A, S, s 0 , G, δ, τ, λ) where:

• L is the set of logical propositions describing the environment.

• A is the set of actions.

• S is the set of state labels.

• s 0 ∈ S is the initial state label.

• G ∈ S is the set of goal state labels.

• δ is the action model.

• τ : S × A → {true, f alse} is the feasibility function.

• λ : S → 2 L is the observation function.

Example 4.1 Consider the Blocksworld example. L is composed of the following propositions: {(ontable red),(ontable green),(ontable blue),(on green blue),. . . }. A possible observed initial state s 0 could be λ(s 0 ) = {(ontable red), (ontable green), (ontable blue), (clear red), (clear green), (clear blue), (handempty)}.

We assume that an observation could be partial and/or noisy. A partial observation is a state where some logical propositions are missing and a noisy observation is a state where the truth value of some propositions is erroneous.

Example 4.2 For example, let take the following noiseless and complete observation: λ(s 0 ) = {(ontable red), (ontable green), (ontable blue), (clear red), (clear green), (clear blue), (handempty)}

A partial observation could be:

λ(s 0 ) = {(ontable green), (ontable blue), (clear green), (handempty)} And a noisy observation could be:

λ(s 0 ) = {(ontable green), (ontable blue), (clear green), (handempty), (on red green)}

A is a set of action labels. Action preconditions, positive and negative effects are given by the functions prec, add and del that are included in δ = (prec, add, del). prec is defined as prec : A → 2 L . The functions add and del are defined in the same way.

Example 4.3 As example, consider the action (pick-up red):

• prec(pick-up red) = {(handempty), (ontable red), (clear red)}

• add(pick-up red) = {(holding red)}

• del(pick-up red) = {(handempty), (ontable red), (clear red)}

The function τ : S × A → {true, f alse} returns whether an action is applicable to a state, i.e. τ(s, a) ⇔ prec(a) ⊆ λ(s). Whenever an action a is applicable in state s i , the state transition function γ : S × A → S returns the resulting state s i+1 = γ(s i , a) such that λ(s i+1 ) = [λ(s i ) \ del(a)] ∪ add(a). A sequence a 1 . . . a n of actions is applicable to a state s 0 when each action a i with 1 ≤ i ≤ n is applicable to the state s i-1 . Given an applicable sequence a 1 . . . a n in state s 0 , γ(s 0 , a 1 . . . a n ) = γ(γ(s 0 , a 1 ), a 2 . . . a n ) = s n . It is important to note that this recursive definition of γ entails the generation of a sequence of states s 0 s 1 . . . s n . A goal state is a state g such that g ∈ G. An action sequence is a solution plan to a planning problem P if and only if it is applicable to s 0 and entails a goal state.

As we have seen in Chapter 3 a formal language is, for a given alphabet A, a subset of A * , i.e. a possibly infinite set of sequences of elements of A. It is therefore possible to represent a STRIPS planning problem as a language such that all the elements of this language is a solution plan. More precisely, we denote L(P) the language generated by the STRIPS planning problem P, this language is defined as follows:

L(P) = {ω = a 1 . . . a n |a i ∈ A, γ(s 0 , ω) |= g}.
We know that the set of languages generated by STRIPS planning problems are regular languages [START_REF] Höller | Assessing the expressivity of planning formalisms through the comparison to formal languages[END_REF] problem P generates a language L(P) that is equivalent to a DFA Σ = (S, A, γ, s 0 , G). S and A are respectively the states and the arcs of the DFA, γ is the transition function, s 0 is the initial state and G is the set of goal state labels. Figure 4.1 gives the DFA for the Blocksworld example.

For any arc a ∈ A, we call pre-set of a the set preset(a) = {s ∈ S | γ(s, a) = s } and post-set of a the set postset(a) = {s ∈ S | γ(s, a) = s }. In our example we have {s 0 , s 7 , s 9 } ∈ preset(pick-up red) and {s 1 , s 13 , s 15 } ∈ postset(pick-up red) (see Figure 4.1).

A STRIPS learning problem is defined as follows: given a set of observations Ω ⊆ L(P), is it possible to learn the DFA Σ, and then infer the action model δ and generalize it into a planning domain ∆ from the learned DFA Σ?

For instance, suppose Ω = { (pick-up red), (put-down red), (pick-up red), (stack red green), (unstack red green), (stack red green) (pick-up blue), (put-down blue), (pick-up red), (stack res blue), (unstack red blue), (put-down red) . . .}

The key idea of our approach is to learn Σ with the action sequences and infers the action model δ : prec(pick-up red) = {(ontable red), (clear red), (handempty)} prec(pick-up blue) = {(ontable blue), (clear blue), (handempty)} . . . ( not ( c l e a r ? x ) ) ( not ( handempty ) ) ( holding ? x ) ) )

STRIPS Learning

The main idea of AMLSI is that it is possible to learn a state machine by testing these transitions and by observing the states resulting from the action executions and to represent it as an action model. AMLSI assumes that it knows the names of the actions, i.e., the names of the transition of the state machine, it can test and it is able to observe the state resulting from their application as a set of logical propositions whose predicates are also known.

Based on these assumptions, AMLSI produces a set of observations Ω by using a random walk and learns as output an action model modeling these observations. The action model learned is expressed as a classical planning domain using the PDDL language. To perform this learning, AMLSI learns first the transition function expressed as a set of actions δ of a particular planning problem P and then, it generalizes δ as a set of planning operators ∆. AMLSI assumes that L, A, S, and s 0 are known, and the observation function λ is possibly partial and noisy. No knowledge of the goal states G is required. As we have no knowledge on G, we consider that the language L(P) is defined for all feasible action sequences ω, even if ω is not a solution plan. Formally,

L(P) = {ω = a 1 . . . a n |a i ∈ A, γ(s 0 , ω) is defined.} AMLSI algorithm consists of 4 steps (see Figure 4.2):
1. Observation Generation. AMLSI produces a set of observations Ω by using a random walk. In Section 4.3.1, we will present how AMLSI is able to efficiently exploit these observations by taking into account not only the fact that some actions are applicable in certain states but also that others are not.

2. DFA Learning. AMLSI learns the DFA Σ representing the observed state transition function γ using an alternative version of the RPNI algorithm [START_REF] Oncina | Inferring regular languages in polynomial updated time[END_REF] (see Section 4.3.2).

PDDL Operator Generation.

Once the DFA Σ is learned, AMLSI infers the action precondition, positive and negative effect functions in δ from the DFA Σ and the observation function λ. Finally, the set of PDDL Operators ∆ are induced from δ (see Section 4.3.3).

Input: Initial State Action Names Observable Predicates

Step I: Observation Generation

Step II: DFA Learning

Step III: PDDL Operator Generation

Step IV: PDDL Operator Refinement I + = { (pick-up red), (put-down red), (pick-up red), (stack red blue), (unstack red blue), (stack red blue) (pick-up blue), (put-down blue), (pick-up red), (stack red blue), (unstack red blue), (put-down reed) (pick-up red), (put-down red), (pick-up blue), (stack blue red), (unstack blue red), (stack blue red) (pick-up blue), (put-down blue), (pick-up blue), (stack blue red), (unstack blue red), (put-down blue) } I -= { (pick-up red), (put-down red), (pick-up red), (stack red blue), (unstack red blue), (pick-up blue) (pick-up red), (put-down red), (pick-up red), (stack red blue), (pick-up blue) (pick-up red), (stack red blue), (put-down red) (pick-up blue), (pick-up red) }

Observation Generation

DFA Learning

The goal of this step is to learn the state machine related to the planning problem. As we have seen in Section 4.2, this state machine can be represented by a regular grammar. Moreover, since we have both positive and negative samples, it is possible to identify this grammar in the limit using RGI algorithms.

Among RGI algorithms, we will use the RPNI1 algorithm [START_REF] Oncina | Inferring regular languages in polynomial updated time[END_REF]. RPNI is very efficient and has all the right properties: (1) it is able to identify in the limit the class of the regular languages; (2) RPNI is locally optimal.

As we have seen before, to identify a regular grammar, we need our sample to be characteristic. It is not possible to construct such a sample a priori. However, if we have a very large sample size, then it is likely that our sample is characteristic; however, one of the conditions for our approach to be effective and to be used in practice is that it requires little training data. We are confronted with a contradiction. To overcome this contraction, we propose to extend the RPNI algorithm using a heuristic. The goal of this heuristic is to allow us to reduce the amount of negative examples. This will allow us to bias the learning of the DFA and to prevent that the DFA from being too general. Indeed, we assume that it is better to have a restrictive DFA, i.e. a DFA that does not recognize all feasible action sequences but limits the recognition of infeasible action sequences, allowing to decrease the risk to learn an action model generating incorrect plans.

Let's consider the definition of an action model. An action is defined using preconditions and effects. These preconditions and effects are defined using a set of logical propositions. These preconditions and effects can be contradictory, i.e. the preconditions of one action will be inconsistent with the effects of another action and vice versa.

Example 4. 4 For example, we have (clear red) ∈ prec(pick-up red) and (clear red) ∈ del(stack green red), so after executing (stack green red), the logical proposition (clear red) will always be absent and the action (pick-up red) will never be feasible, no matter what action precedes action (stack green red). In practice, very few actions follow each other. For our Blocksworld problem, the action (stack green red) can only be followed by the actions (unstack green red) and (pick-up blue).

We will use this property to reduce our negative sample size. More precisely, we propose to force the DFA learned to accept only the sequences defined in I + and reject all other unobserved sequences. A simple way to achieve this result is to compute all unobserved pairwise sequences (PS) (a i a j ) and add them to the set of negative examples I -. The computation of the set of unobserved PS consists in computing all the possible action pairs from the actions A of the DFA and to subtract the paris present in I + . Formally, the set of unobserved PS added to I - is defined as follows:

{(a i , a j ) | (a i , a j ) ∈ A 2 and ω ∈ I + s.t. ω = ω 1 , a i , a j , ω 2 } (4.1)
Example 4.5 For example, the pairwise sequence ((pick-up red), (pick-up blue)) will be added to I -because ((pick-up red), (pick-up b)) is not a subsequence contained in I + . In contrast, the pairwise sequence ((pick-up red), (put-down red)) will not be added to I - because ((pick-up red), (put-down red)) is a subsequence of the sequence (pick-up red), (put-down red), (pick-up red), (stack red blue), (unstack red blue), (stack red blue) contained in I + .

Example 4.6 Figure 4.4 compares the execution of the vanilla RPNI algorithm and RPNI improved with PS. We can note that the unobserved sequences prevent the merging of some states. For instance, the merge between states 0 and 1 is no more possible because the sequence (pick-up red), (pick-up blue) have been added to I - based on Equation 4.1. Therefore, the DFA learned with this improvement accepts only observed sequences decreasing the risk to learn an action model generating incorrect plans. Note that the unobserved sequences are only used for improving the DFA learning step. For the next steps, we assume that I -contains only infeasible observed sequences.

Operator generation

Operator generation consists in generating δ = (prec, add, del) from the learned DFA and generalizes it into a set of PDDL operators ∆. Operator generation is based on three steps: Precondition generation To learn the preconditions prec(a) of the action a, AMLSI computes the logical propositions that appear in all the states preceding a in Σ:

prec(a) = ∩ s∈preset(a) λ(s) (4.2)
Example 4.7 Let take the DFA in Figure 4.1. We have {s 0 , s 7 , s 9 } ∈ preset(pick-up red). Suppose we have the following, possibly partial and/or noisy, observations:

• λ(s 0 ) = {(ontable red), (clear red), (handempty) } • λ(s 7 ) = {(ontable red), (clear red) } • λ(s 9 ) = {(ontable red), (handempty) }
Then, the precondition is computed as follows: prec(pick-up red) = {(ontable red)}.

Effect generation

To learn the positive effects add(a) and the negative effects del(a) of an action a, AMLSI computes the logical propositions that never appear in states before the execution of a, and always present after a execution:

add(a) = ∩ s∈postset(a) λ(s) \ prec(a) (4.3) Symmetrically, del(a) = prec(a) \ ∩ s ∈postset(a) λ(s) (4.4)
Example 4.8 Let take the DFA in Figure 4.1. We have {s 1 , s 13 , s 15 } ∈ postset(pick-up red). Suppose we have the following, possibly partial and/or noisy, observations:

• λ(s 1 ) = {(holding red) } • λ(s 13 ) = {(holding red) } • λ(s 15 ) = {(holding red) }
Then, the effects is computed as follows: add(pick-up red) = {(holding red)} and del(pick-up red) = {(ontable red)}.

Action generalization Once actions' preconditions and effects have been learned, AMLSI generalizes actions to operators. To do this, AMLSI breaks down the DFA actions into sets of actions that have the same name, number and type of parameters. Each of these sets of actions can be generalized into an operator.

Example 4.9 For example, assumes that we have two actions (pick-up red) and (pickup green) that have the same name, number and type of parameters in the learned DFA.

The action (pick-up red) is defined as follows:

prec(pick-up red) = {(ontable red)}. add(pick-up red) = {(holding red)}. del(pick-up red) = {(ontable red)}. (4.5)
and the action (pick-up green) is defined as follows:

prec(pick-up green) = {(ontable green), (clear green)}. add(pick-up green) = {(holding green)}. del(pick-up green) = {(ontable green), (clear green)}.

(4.6)

These two actions can be generalized into one operator (pick-up ?x) by using the OIsubsumption (subsumption under Object Identity) [START_REF] Esposito | Multistrategy theory revision: Induction and abduction in INTHELEX[END_REF]. The OIsubsumption consists in substituting constants in preconditions and effects by variables by respecting the type of the parameters. In our example, the subsumption of the preconditions of (pick-up red) is {(ontable ?x)} and the subsumption of the preconditions of (pick-up green) is {(clear ?x), (ontable ?x)}.

Then, to compute operators preconditions and effects AMLSI computes the less general preconditions and effects satisfied for all subsumptions. In the end, the operator (pick-up ?x) obtained is : This generalization method applied for each set of actions that have the same name, number and type of parameters in the learned DFA allows us to ensure that all the necessary preconditions, i.e., the preconditions allowing to differentiate the states where an action is feasible from a state where an action is infeasible, to be all encoded.

We can note that the learned operator is not correct. Indeed, some preconditions and effects are missing. This due to some observations being partial. To deal with it, we perform a refinement step.

Operator refinement

Due to the partial and noisy observations, AMLSI performs a refinement step. This refinement is carried out in three substeps. The first two substeps are to refine the preconditions and effects to deal with the partial observations and ensure that the learned operators are able to regenerate the induced DFA. Specifically, these two steps ensure that operators can regenerate each DFA transition. The last substep is to refine the operators to deal with noisy observations using a Tabu Search.

Effect refinement This step ensures that the generated operators allow to regenerate the induced regular grammar. We use the observation function to check that for each pair of consecutive actions a and a in the DFA, the effects of action a applied in the state s satisfy the preconditions of action a . If it is not the case, we add in the effects of a the propositions satisfying the preconditions of a .

Example 4.10 For example, let take the action (stack red green) is an outcoming edge of the state s 1 (see Figure 4.1) and an incoming edge of the state s 4 , and the action (unstack red green) is an outcoming edge of the state s 1 . Likewise suppose (on red green) ∈ λ(s 1 ) and (on red green) ∈ prec(unstack red green), we put (on ?x ?y) ∈ add(stack ?x ?y) to ensure that the action (unstack red green) is feasible in state s 4 after applying (stack red green) in state s 1 .

Precondition refinement

In this step, we assume like [START_REF] Yang | Learning action models from plan examples using weighted MAX-SAT[END_REF] that the propositions of the negative effects of an operator must be in its preconditions. Thus, for each negative effect in an operator, we add the corresponding predicate in its preconditions.

Example 4.11 For example, if (holding ?x) ∈ del(stack ?x ?y) then we put (holding ?x) ∈ prec(stack ?x ?y).

Moreover, since effect refinements depend on preconditions and precondition refinements depend on effects, we repeat these two steps until convergence, i.e., no more precondition or effect is added. The refinement process converges because (1) the preconditions and the effects can be added only once during the whole refinement process of an operator and ( 2) the number of preconditions and effects that could be added is limited by the number of effects and preconditions of the next action in the DFA. Tabu Search The refinement step previously described is able to find most of the preconditions and the effects of the operators even with partial observations. However, this refinement does not prevent to remove relevant or add irrelevant preconditions and effects when observations are noisy. For example, suppose that the action (stack green blue) is applicable in two states s and s in the DFA, i.e., s and s ∈ preset(stack green blue). Now, suppose that the observation function λ(s) returns (clear blue) as true and λ(s ) returns (clear blue) as f alse due to the noise. In that case, (clear blue) is not included in the preconditions of (stack green blue) even if it has to. Thus, after generalisation, the operator (stack ?x ?y) will not have as precondition (clear ?y).

To deal with this problem, we propose to use Tabu Search [START_REF] Glover | Tabu Search[END_REF]. Tabu Search is a classical meta-heuristic search method employing local search methods used for mathematical optimization. The idea is to explore variants of the operators set learned in the previous step by adding and removing preconditions and effects. At each steps only variants improving the set of learned operators are kept until a local minimum is found. To determine whether one variant is better than another it is necessary to define an evaluation function. This evaluation function is called a fitness function. In practice, a variant ∆ of an operators set is better than an other one given a positive and a negative set of observations if : (1) ∆ accepts more positive observations, (2) ∆ rejects more negative observations and ( 3) the state sequences produced by applying the transition function γ ∆ on the positive actions sequences observed in I + violate fewer preconditions and effects than the sequences produced with the other. Formally, the fitness function used by AMLSI to evaluate a candidate variant ∆ given the observations sets I + and I - is defined as follows: To conclude, the complexity to compute the fitness score for all candidate variants

f (∆, I + , I -) = ∑ ω∈I + accept(∆, w) + ∑ ω∈I - reject(∆, w) + ∑ w∈I + ∑ s∈γ ∆ (s 0 ,ω) |s ∩ λ(s)| -|s\λ(s)| (4.7) where: accept(∆, w) = 1 if ∀s ∈ γ ∆ (s 0 , w) s ∈ λ(s) 0 otherwise. (4.8) reject(∆, w) = 1 if ∃s ∈ γ ∆ (s 0 , w) s ∈ λ(s) 0 otherwise. ( 4
∆ is O(|A|.|S| 2 .(|I + | + |I -|)).
Once the Tabu Search is done, i.e. a local optimum of the fitness score f (∆, I + , I -) is reached, we repeat all the three refinement steps (effect and precondition refinement plus Tabu Search) until convergence (see Figure 4.5).

Effects Preconditions

Formal properties

In this section we will give the formal properties of the AMLSI algorithm. First, we will show that the DFA Learning algorithm identifies the language L(P) in the limit.

Then, we will show that STRIPS Domain Learning is sound. STRIPS domain learning is sound means that, for a given observation set Ω, AMLSI learns a STRIPS Domain generating the given observation set. We will prove the STRIPS Domain Learning soundness when observations are complete and noiseless. When observations are partial and/or noisy it is possible that the learning is not sound. However, we will show experimentally in Section 4.5 that AMLSI keeps high performances when observations are partial and/or noisy despite the lack of soundness.

DFA Learning Formal Properties

We know that the RPNI algorithm identifies the class of regular language in the limit [START_REF] Dupont | Incremental regular inference[END_REF]. As the language L(P) is regular, RPNI algorithm identifies this language. So, there exists a sample Ω c = {I c + , I c -} such that:

• RPN I(I c + , I c -) → Σ with L(Σ) ≡ L(P). • ∀I + , I -such that ∀ω ∈ I + ω ∈ L(P) and ∀ω ∈ I -ω ∈ L(P) then RPN I(I c + , I c -) ≡ RPN I({I c + ∪ I + }, {I c -∪ I -}).
The sample Ω c is the characteristic sample of L(P). See Chapter 3 for more details.

As we have seen in Section 4.3.2, the RPNI algorithm is improved with unobserved sequences. More precisely, AMLSI adds in the negative sample Iall unobserved pairwise sequences (a i a j ). Let I PS -be the set of unobserved pairwise sequences.

Property 4.1 RPN I(I c + , I c -) ≡ RPN I(I c + , {I c -∪ I PS -}) if and only if ∀ω ∈ I PS -ω ∈ L(P) I PS
-contains all pairwise sequences (a i a j ) such that: (a i , a j ) ∈ A 2 and ω ∈ I c + s.t. ω = (ω 1 , a i , a j , ω 2 )}. So, to ensure that the property 4.1 is checked we need that all pairwise sequences present in the language L(P) are present in the positive sample I c + . Formally:

∀(a i , a j ) ∈ A 2 s.t. ∃ω = (ω 1 , a i , a j , ω 2 ) ∈ L(P) =⇒ ∃ω = (ω 1 , a i , a j , ω 2 ) ∈ I c + (4.10)
Theorem 4.1 DFA Learning algorithm identifies the language L(P) in the limit.

Proof 4.4 AMLSI learns Σ using the RPNI algorithm improved with unobserved sequence. We know that RPNI identifies int the limit L(P). Also, thanks the property 4.1, we know that RPNI improved with unobserved sequence identifies in the limit L(P) if and only if the positive sample satisfies the constraint given by Equation 4.10. As there exists a characteristic sample for the language L(P) then the DFA Learning algorithm identifies the language L(P) in the limit.

Domain Learning Soundness

We will now show that the PDDL domain learned by AMLSI, when L(Σ) ≡ L(P) and when observations are complete and noiseless, is a PDDL domain ∆ P able to generate L(P) and the observations function λ. AMLSI returns the PDDL domain refined by the refinement steps (see Section 4.3.4). We must therefore show that the different refinement steps allow to learn a PDDL Domain ∆ such that ∆ is able to generate L(P) and the observations function λ. More precisely, we will prove that when the Tabu Search reaches the global maxima, AMLSI refinement stops and returns a PDDL domain able to generate L(P) and the observations function λ.

Let's us assume there exists a PDDL Domain ∆ P such that ∆ P is able to generate the regular language L(P) and the observation function λ. Lemma 4.1 If L(Σ) ≡ L(P), when observations are complete and noiseless, ∆ P maximizes the fitness function f (∆, I + , I -) of the Tabu Search. Proof 4.5 First of all, as ∆ P generates the regular language L(P) we have:

∆ P = argmax ∆ ( ∑ ω∈I + accept(∆, w)) (4.11) ∆ P = argmax ∆ ( ∑ ω∈I - reject(∆, w)) (4.12)
Indeed, if ∆ P generates the regular language L(P) then ∀ω ∈ I + , ∆ P can generate ω. So,

∑ ω∈I + accept(∆, w) = |I + | = Max ∆ ( ∑ ω∈I + accept(∆, w)).
In the same way, if ∆ P generates the regular language L(P) then ∀ω ∈ I -, ∆ P cannot generate ω. So,

∑ ω∈I - reject(∆, w) = |I -| = Max ∆ ( ∑ ω∈I - reject(∆, w)).
Then, as ∆ P generates the observation function λ we have:

∆ P = argmax ∆ ( ∑ w∈I + ∑ s∈γ ∆ (s 0 ,ω) |s ∩ λ(s)| -|s\λ(s)|) (4.13) Indeed, if ∆ P generates the observation function λ then ∑ w∈I + ∑ s∈γ ∆ P (s 0 ,ω) |s ∩ λ(s)| - |s\λ(s)| = ∑ w∈I + ∑ s∈γ ∆ P (s 0 ,ω) |λ(s)| -0 = Max ∆ ( ∑ w∈I + ∑ s∈γ ∆ (s 0 ,ω) |s ∩ λ(s)| -|s\λ(s)|)
Finally, we have

Max ∆ ( f (∆, I + , I -)) = Max ∆ ( ∑ ω∈I+ accept(∆, w) + ∑ ω∈I- reject(∆, w) + ∑ w∈I+ ∑ s∈γ ∆ (s0,ω) |s ∩ λ(s)| -|s\λ(s)|) = Max ∆ ( ∑ ω∈I+ accept(∆, w)) + Max ∆ ( ∑ ω∈I- reject(∆, w)) + Max ∆ ( ∑ w∈I+ ∑ s∈γ ∆ (s0,ω) |s ∩ λ(s)| -|s\λ(s)|)) (4.14)
And thanks to Equations 4.11,4.12 and 4.13 we know that f (∆ P ,

I + , I -) = Max ∆ ( f (∆, I + , I -)).
+ ∈ E + (resp. ω -∈ E -).

Experiments

Experimental setup

Our experiments are based on 13 STRIPS-Compliant IPC2 benchmarks: Blocksworld, Gripper, Hanoi, N-Puzzle, Peg Solitaire, Parking, Zenotravel, Sokoban, Visit All, Elevator, Spanner, Logistics and Floortile. Table 4.1 shows our experimental setup 3 . We test each IPC action model with 3 different instances over ten runs, and we use ten randomly generated seeds for each run. Also, we generate partial observations by randomly removing a fraction of the propositions of the states, and we generate noise by changing the value of a fraction of the observable propositions. All tests were performed on an Ubuntu 14.04 server with a multicore Intel Xeon CPU E5-2630 clocked at 2.30 GHz with 16GB of memory. PDDL4J library [START_REF] Pellier | Pddl4j: a planning domain description library for java[END_REF] was used to generate the benchmark data.

Evaluation metrics

Three metrics are used for the evaluation: the syntactical error [START_REF] Zhuo | Learning complex action models with quantifiers and logical implications[END_REF] that computes the distance between the original action model and the action model learned, the accuracy (Zhuo et al., 2013) that expresses the capability of the action model learned to solve new problems (without proofreading). Even though the syntactical error is the most used metric in the literature, we argue that the accuracy is the most important metric in practice for planning because it measures to what extent a learned action model is useful. Indeed, it often happens that one missing precondition or effect, which amounts to a small syntactical error, makes the learned action model unable to solve planning problems. Finally, the last metric is the FScore (van Rijsbergen, 1979) that expresses the capability of the learned action model to generate the grammar related to the planning problem.

Formally, the syntactical error error(o) for an operator is the Hamming distance between the learned operator and the ground truth operator, i.e. the number of extra or missing predicates in the preconditions prec(o), the positive effects add(o) and the negative effects del(o) divided by the total number of possible predicates. By extension, the syntactical error for a action model composed of a set of operator O is:

E σ = 1 |O| ∑ o∈O error(o)
Then, FScore = 2.P.R P+R where R is the recall, i. Finally, the accuracy Acc = N N * is the ratio between N, the number of correctly solved problems with the learned action model, and N * , the total number of problems to solve. In the rest of this section the accuracy is computed over 20 problems. STRIPS problems are solved with Fast Downward v19.06 [START_REF] Helmert | The Fast Downward planning system[END_REF]. For each experiment, plan validation is done with VAL [START_REF] Howey | Val's progress: The automatic validation tool for pddl2. 1 used in the international planning competition[END_REF], which is used in the IPC competitions. In addition to the Accuracy we report the IPC score in order to compute the quality of generated plans. The score of a action model on a solved problem is the ratio between the length of a reference plan, i.e. a plan generated by the reference IPC action model, and the length of the plan generated by the learned action model. The score on an unsolved problem is 0. The score of a learned action model is the sum of its scores for all problems. 4.6, 4.7, 4.8 shows the average performance of AMLSI and LSONIO obtained on the 13 action models of our benchmarks when varying the training dataset size. The size of the training set is indicated in number of actions. AMLSI and LSONIO are tested with 20 experimental scenarios: the level of observability varies between 20% and 100% and the level of noise varies between 0% and 20%. We have three variants of AMLSI: (B) Base: DFA learning is done without Pairwise Sequences (PS) and without Tabu Search, (B+PS) Base + PS: DFA learning is done with PS but without Tabu Search, and (B+PS+Tabu) Base + PS + Tabu: DFA learning is done with PS and with Tabu Search during the refinement step.

Discussion

Comparison with LSONIO We observe that AMLSI outperforms LSONIO whatever the size of the learning dataset in terms of accuracy or in terms of syntactical distance. We also observe that AMLSI needs very little data to obtain a relatively large accuracy (almost 70% with only a learning dataset of 400 actions) in the most difficult scenario.

Ablation study

The Base+PS variant is more robust to partial observations than the Base variant of AMLSI. This is due to the fact that DFA learned with PS are generally better than action model learned without PS. More precisely, DFA learned with PS generally have fewer states and fewer transitions. This allows for fewer false transitions which makes it easier to learn effects and preconditions. However, when observations are noisy, the Base+PS variant is not able to learn action models accurate enough to be used for planning whatever the level of observability. Only the Base+PS+Tabu variant is both robust to partial and noisy observations. Our ablation study confirms that adding unobserved Pairwise Sequences improves the learning of the DFA, and makes AMLSI more robust to partial observations while refining the preconditions and the effects by using a Tabu Search allows AMLSI to learn accurate action models with a high level of noise.

Finally, we observe that the plans generated with the learned action models are generally longer than plans generated with the IPC action models, even with optimal accuracy (ipc < 20). When the accuracy is not optimal, we notice that the IPC score is close to the ratio between the IPC score with optimal accuracy and the rate of solved problems, this implies that even when the accuracy is not optimal the plans are not much longer than the original plans.

Conclusion

In this chapter, we have presented the AMLSI approach. The AMLSI approach learns action models from interactions with the environment in which the agent will have to solve planning problems. The AMLSI approach is divided in four steps: (1) AMLSI generates a set of observations, (2) AMLSI learns the DFA related to the planning problem using a variant the RPNI algorithm, (3) AMLSI learns from the DFA the action model and express it as a PDDL planning domain and ( 4) AMLSI refines the action model. Also, we have given the formal properties of our approach. More precisely, we have shown that our variant of the RPNI algorithm preserves the identification in the limit property of the vanilla RPNI algorithm. Also, we have shown the soundness of the AMLSI approach.

Moreover, we can draw several lessons from our experimentation. First, we show that the AMLSI approach is highly accurate whatever the level of partiality and noise of the observations. Moreover, we have seen that AMLSI requires little training dataset. Also, we have shown that AMLSI outperforms LSONIO, the closest state-of-the-art approach. Finally, thanks to the ablation study, we were able to show the usefulness of each step of our approach.

While these results are encouraging, the AMLSI approach has several limitations. First of all, we can observe that AMLSI requires little training dataset to be accurate. Also, the larger the data the better the results. However, as we have seen in the previous part, acquiring data is a difficult and costly process. Although AMLSI requires little training data, it is not possible to know a priori how much training data is required to learn a model. It is therefore important to be able to incrementally learn the action model and to have a criterion to know that the action model is learned and to stop the learning process. This is what we will see in the next chapter. Also, the AMLSI approach learns STRIPS action models, but we have seen in the previous part that STRIPS action models are based on assumptions that are too restrictive to be used in realworld applications. In the following chapters, we will propose two extensions for AMLSI to learn less restrictive action models.

Introduction

In the previous chapter, we have presented the AMLSI approach. We have seen that AMLSI successfully address the accuracy issue even when observations are partial and/or noisy. Also, we have seen that AMLSI requires few data to be accurate. However, in practise, data acquisition is a long term evolutive process: in real world applications, training data become available gradually over time, are difficult and costly to obtain, as for instance, Mars Exploration Rover operations [START_REF] Bresina | Activity planning for the mars exploration rovers[END_REF] or robot fleets for offshore missions [START_REF] Carreno | Task allocation strategy for heterogeneous robot teams in offshore missions[END_REF]. Moreover, in practice, it is important to be able to update learned action models to new incoming data without restarting the learning process from scratch. Finally, it is also important to know when to stop learning in order to know when to stop the data acquisition process and minimize the amount of data acquired: a convergence criterion is required. In this chapter, we propose an incremental version of AMLSI called IncrAMLSI [START_REF] Grand | Incramlsi: Incremental learning of accurate planning domains from partial and noisy observations[END_REF] to overcome these issues.

The rest of this chapter is organized as follows. In Section 5.2 we detail the incremental learning process and give the formal properties of this incremental extension. In Section 5.3 we show that our incremental allows to efficiently consume training data. The problem statement being the same as in the previous chapter, we will not present it in this chapter.

Incremental Learning

An overview of IncrAMLSI is shown in Figure 5.1. IncrAMLSI learns incrementally the action model from incoming data and does not restart from scratch when new data become available at each iteration t. More precisely, IncrAMLSI consists in incrementally updating the PDDL domain ∆ t with the new incoming training datasets Ω t1 available at iteration t to produce the PDDL domain ∆ t+1 . This algorithm is made up of three steps: The incremental process is operated each time new training datasets are input and until convergence of the PDDL domain.

Operator Overhaul

When at iteration t new positive and negative datasets are integrated and the DFA is updated, it is possible that the domain previously learned ∆ t is no longer compatible with it. There are two possibilities:

1. Some operators may not have been generated in the previous datasets, so IncrAMLSI have to add them 2. Some effects and preconditions in ∆ t have to be removed to match the updated DFA Algorithm 2 describes the procedure to compute ∆ t+1 from ∆ t , I t + , and I t -at iteration t. First of all, IncrAMLSI generates new operators in a PDDL domain ∆ t new (line 1). Then, IncrAMLSI removes all extra preconditions and effects in ∆ t (line 2 -10). Finally, IncrAMLSI merges ∆ t new with ∆ t (line 11) by making the union sets of the preconditions and the effects of the operators.

First of all, functions backtrackInfeasibleEffects and backtrackInfeasiblePreconditions (line 2, 3) remove the effects and preconditions infeasible in the DFA. More precisely, backtrackInfeasibleEffects removes positive and negative effects that does not respect the conditions given by Example 5.1 For example, suppose we have (holding ?x) ∈ add(unstack ?x ?y), and there are no states in the current DFA where this effect appears, then (holding ?x) is removed.

Likewise, backtrackInfeasiblePreconditions removes extra preconditions that does not validate Equation 4.2.

Example 5.2 For example, suppose we have (holding ?y) ∈ prec(unstack ?x ?y) and ∃s ∈ µ preset (unstack green blue) such that (holding blue) ∈ λ(s), then (holding ?y) is removed.

Then, as observed states are noisy and partial, it is possible that some extra preconditions and effects are not present in the current DFA. Therefore lines 4 -10 remove extra preconditions and effects independently of this DFA. First of all, backtrackPreconditions (line 6) removes all the preconditions that are not compatible with I t + .

Example 5.3 For instance, suppose we have this positive sample:

(pick-up green), (put-down green), (pick-up green), (stack green blue), (unstack green blue) ∈ I t + and, when we execute it with the current domain ∆ t we have:

(holding green) ∈ λ(γ(s 0 , (pick-up green), (put-down green), (pick-up green), (stack green blue) ))

i.e. in the state before the last action (unstack green blue), the proposition (holding green) is absent and suppose that (holding ?x) ∈ prec(unstack ?x ?y). Thus we remove this unsatisfied precondition (holding ?x) from the operator (unstack ?x ?y).

Then, with backtrackNegativeEffects (line 7), we remove all the negative effects that does not satisfy PDDL syntactical constraints [START_REF] Yang | Learning action models from plan examples using weighted MAX-SAT[END_REF]) (e.g., negative effects must be in the preconditions etc.) We repeat these two functions until the domain accepts all the positive samples (line 8), i.e. the domain is able to regenerate all the positive samples I t + . The next step (line 9) in the algorithm is to remove all the effects that are not compatible with the negative samples I t -with function backtrackEffects.

Example 5.4 Suppose that two samples share the same prefix (unstack green blue):

(unstack green blue), (put-down green) ∈ I t + .

and

(unstack green blue), (pick-up green) ∈ I t -. Also, suppose that, wrongly, (holding ?y) ∈ add(unstack ?x ?y), and, correctly, (holding ?x) ∈ prec(put-down ?x). And suppose that the current domain ∆ t can generate (unstack green blue) . The extra effect (holding green) ∈ add(unstack green blue) allows to the precondition of the unfeasible action (put-down blue) to be satisfied, and (holding blue) is not in the preconditions of the feasible action (put-down green). Thus, we remove the effect (holding ?y) from the operator (unstack ?x ?y).

As the backtrackEffects function removes effects, it is possible that some positive samples can no longer be generated by the current domain ∆ t . We therefore have to repeat these three backtracking functions until the domain accepts all the positive samples (line 10). Indeed, repeating these functions until the domain accepts all the positive samples allows to ensure that the last removed effects do not impact the acceptation of the positive samples (termination is ensured by only allowing precondition and/or effect withdrawals).

Convergence

IncrAMLSI stops when, after a given number T of iterations, the domain ∆ t checks three conditions: (1) ∆ t accepts all the positive samples I t + , (2) ∆ t rejects all the negative samples I t -, and (3) ∆ t = ∆ t+1 during T iterations. We will show experimentally (see section 5.3) that IncrAMLSI converges with partial and noisy observations. However, it is possible to show formally that it is true when state observations are complete and noiseless and T → ∞. More precisely, we can prove that IncrAMLSI converges to a domain ∆ P able to generate the language L(P) and the observation function λ (see Chapter 4). Lemma 5.1 When observation are complete and noiseless and when the observation set Ω t = {I t + , I t -} is characteristic, then the overhaul operators step returns a domain ∆ t+1 with no extra preconditions and effects w.r.t the domain ∆ P .

Proof 5.1 First of all, as the observation set is characteristic we know, thanks to Theorem 4.1 (see Chapter 4), that L(Σ t ) = L(P). Also, as observations are complete and noiseless, ∆ t new does not contain any extra preconditions and effects. Then, as functions backtrackInfeasibleEffects and backtrackInfeasiblePreconditions remove all preconditions and effects incompatible with equations of the operators generation steps, then all extra preconditions and effects are removes. Finally, as all extra preconditions and effects are removed and as ∆ t new does not contain any extra preconditions and effects then ∆ t+1 does not contain any extra preconditions and effects Theorem 5.1 When observation are complete and noiseless and when T → ∞, IncrAMLSI converges to ∆ P . Proof 5.2 First of all, if T → ∞ then the observation set Ω T is characteristic. Also, thanks the Lemma 5.1 we know that, before refinement, the domain ∆ T does not contain any extra preconditions and effects. Finally, thanks the Theorem 4. is characteristic and when observations are complete and noiseless. So, we can conclude that IncrAMLSI converges to ∆ P when observation are complete and noiseless and when T → ∞.

Experiments

Experimental setup

Our experiments are based on 13 STRIPS-Compliant IPC 2 benchmarks: Blocksworld, Gripper, Hanoi, N-Puzzle, Peg Solitaire, Parking, Zenotravel, Sokoban, Visit All, Elevator, Spanner, Logistics and Floortile. Table -5.1 shows our experimental setup 3 .

We test each IPC domain with 3 different initial states over ten runs, and we use randomly generated seeds for each run. Also, we generate partial observations by randomly removing a fraction of the propositions of the states, and we generate noise by changing the value of a fraction of the observable propositions. All tests were performed on an Ubuntu 14.04 server with a multicore Intel Xeon CPU E5-2630 clocked at 2.30 GHz with 16GB of memory. PDDL4J library [START_REF] Pellier | Pddl4j: a planning domain description library for java[END_REF] was used to generate the benchmark data.

Evaluation metrics

The metrics used for this evaluation is the accuracy (Zhuo et al., 2013) that measures the learned domain performance to solve new problems. Formally, the accuracy Acc = N N * is the ratio between N, the number of correctly solved problems with the learned domain, and N * , the total number of problems to solve. In the rest of this section the accuracy is computed over 20 problems. The problems are solved with Fast Downward v19.06 [START_REF] Helmert | The Fast Downward planning system[END_REF]. Plan validation is done with VAL [START_REF] Howey | Val's progress: The automatic validation tool for pddl2. 1 used in the international planning competition[END_REF], which is used in the IPC competitions. 4 IncrAMLSI is tested with 20 experimental scenarios: the level of observability varies between 20% and 100% and the level of noise varies between 0% and 20% Whatever the experimental scenario, IncrAMLSI learns from accurate models. Also, increasing T generally leads to better results; Finally, we can observe that whatever the value of T and the experimental scenario, IncrAMLSI converges in less than 35 iterations.

Discussion

Conclusion

In this chapter, we have presented the incremental extension of the AMLSI approach: IncrAMLSI. IncrAMLSI learns incrementally action models. The incremental learning process is divided in 3 steps: each time new training data is received, (1) IncrAMLSI updates the DFA, ( 2) removes extra preconditions and effects and ( 3) refines the action model. IncrAMLSI stops when the convergence criterion is reached. In addition, we have shown the convergence of the IncrAMLSI extension.

The AMLSI approach, and its incremental extension, learns STRIPS action models, but we have seen in the previous part that STRIPS action models are based on assumptions that are too restrictive to be used in real-world applications. In the following chapters, we will propose two extensions for AMLSI to learn less restrictive action models. More precisely, we will extend the AMLSI approach to learn temporal action model (see Chapter 6) and HTN action models (see chapter 7).

Introduction

As seen previously, hand-encoding and proofreading STRIPS action models is difficult, and this is even harder with less restrictive action models such as temporal action models. It is therefore essential to develop tools allowing to automatically learn temporal action models. As we have seen in Chapter 2 several approaches have been proposed to automatically learn STRIPS action models: ARMS [START_REF] Yang | Learning action models from plan examples using weighted MAX-SAT[END_REF], SLAF [START_REF] Shahaf | Learning partially observable action schemas[END_REF], Louga [START_REF] Kucera | LOUGA: learning planning operators using genetic algorithms[END_REF], LSONIO [START_REF] Mourão | Learning STRIPS operators from noisy and incomplete observations[END_REF], LOCM [START_REF] Cresswell | Acquiring planning domain models using LOCM[END_REF], IRale (Rodrigues et al., 2010a), PlanMilner [START_REF] Segura-Muros | Learning numerical action models from noisy and partially observable states by means of inductive rule learning techniques[END_REF].

A major open issue is to learn temporal action models [START_REF] Fox | PDDL2. 1: An extension to PDDL for expressing temporal planning domains[END_REF]. Temporal action models are action models allowing to represent durative actions, i.e. actions that have a duration, and whose preconditions and effects must be satisfied and applied at different times. An important property of durative actions is that they can be executed concurrently. Temporal action models have different levels of action concurrency [START_REF] Cushing | When is temporal planning really temporal?[END_REF]. Some are sequential, which means that all the plan parts containing overlapping durative actions can be rescheduled into a completely sequential succession of durative actions: each durative action starts after the previous durative action is terminated. One important property of sequential temporal action models is that they can be rewritten as non-temporal action models, and therefore used by classical planners. Some temporal action models require other forms of action concurrency such as Single Hard Envelope (SHE) [START_REF] Coles | Managing concurrency in temporal planning using planner-scheduler interaction[END_REF]. SHE is a form of action concurrency where a durative action can be executed only if another durative action called the envelope extends over it. This is due to the need by the enveloped durative action of a resource, all along its execution, added at the start of the envelope and deleted at the end of the envelope. One important property of SHE temporal action models is that they cannot be sequentially rescheduled. Although some approaches have been proposed to learn temporal features [START_REF] Gabel | Online inference and enforcement of temporal properties[END_REF][START_REF] Neider | Learning linear temporal properties[END_REF][START_REF] Gaglione | Learning linear temporal properties from noisy data: A maxsat-based approach[END_REF][START_REF] Shah | Bayesian inference of temporal task specifications from demonstrations[END_REF], only [START_REF] Garrido | Learning temporal action models via constraint programming[END_REF] proposed an approach learning temporal action models. However this approach is limited to sequential temporal action models. To our best knowledge, there is no learning approach for both SHE and sequential temporal action models.

In this chapter, we present TempAMLSI [START_REF] Grand | Tempamlsi: Temporal action model learning based on STRIPS translation[END_REF], an accurate learning algorithm for both SHE and sequential temporal action models. TempAMLSI is built on AMLSI (see Chapter 4). Some planners (Fox and Long, 2002a;[START_REF] Halsey | Crikey-a temporal planner looking at the integration of scheduling and planning[END_REF][START_REF] Celorrio | Temporal planning with required concurrency using classical planning[END_REF][START_REF] Blanco | Forward-search temporal planning with simultaneous events[END_REF] solve temporal planning problems by using non-temporal planners and translation techniques. The key idea of the TempAMLSI approach is to reuse these translation techniques for the learning problem. Like AMLSI, TempAMLSI interacts with the environment to generate input feasible and infeasible action sequences to frame what is allowed by the targeted action model. Then, the temporal learning consists of three steps: (1) TempAMLSI translates temporal sequences into STRIPS sequences, (2) TempAMLSI learns a non-temporal action model with AMLSI, and then (3) translates it into a temporal action model.

TempAMLSI contributions in temporal action model learning are threefold:

• Concurrency: TempAMLSI is able to learn both sequential and SHE temporal action models,

• Partial and noisy observations: TempAMLSI is able to learn temporal action models with both partial and noisy observations.

• Accuracy: TempAMLSI is accurate even with highly partial and noisy learning datasets: thus, it minimises proofreading for AI planning experts.

We show that in many temporal benchmarks TempAMLSI does not require any correction of the learned domains at all.

The rest of this chapter is organized as follows. In Section 6.2 we present a problem statement. In Section 6.3 we give some backgrounds on STRIPS translation techniques, in Section 6.4, we detail TempAMLSI steps. Finally, Section 6.5 evaluates the performance of TempAMLSI on IPC temporal benchmarks.

Problem Statement

We propose a formal framework inspired by [START_REF] Höller | Translating totally ordered htn planning problems to classical planning problems using regular approximation of context-free languages[END_REF] in order to define the temporal learning problem. This formal framework extends the formal framework proposed in Chapter 2 to the learning problem. Definition 6.1 A temporal planning problem P is a tuple (L, A, S, d, s 0 , G, δ, τ, λ) where:

• L is the set of logical propositions describing the environment.

• A is the set of durative actions.

• S is the set of state labels.

• d : A → R is the duration function.

• s 0 ∈ S is the initial state label.

• G ∈ S is the set of goal state labels.

• δ is the temporal action model.

• τ : S × A → {true, f alse} is the feasibility function.

• λ : S → 2 L is the observation function.

As for STRIPS problems, L is a set of logical propositions, S is a set of states, s 0 ∈ S is the initial state, G is the set of goal states, and λ is the observation function. A is a set of durative actions and d : A → R is the duration function. Unlike STRIPS planning problems, action preconditions, positive and negative effects are labeled with time labels at-start, at-end and overall. More precisely, δ includes:

• prec : A × {s, o, e} → 2 L : preconditions of a ∈ A at start, over all, and at end, respectively.

• add : A × {s, e} → 2 L : positive effects of a ∈ A at start and at end, respectively. • del : A × {s, e} → 2 L : negative effects of a ∈ A at start and at end, respectively.

Example 6.1 For example, for the the action (stack green blue right) we have:

• Preconditions:

• prec((stack green blue right),o) = {(holding green right)}.

• prec((stack green blue right),s) = ∅.

• prec((stack green blue right),e) = {(clear blue)}.

• Positive effects:

• add((stack green blue right),e) = {(on green blue), (clear green)}.

• add((stack green blue right),s) = ∅

• Negative effects:

• del((stack green blue right),e) = {(clear blue), (holding green right)}.

• del((stack green blue right),s) = ∅

The semantics of durative actions is defined in terms of two discrete events start-a and end-a, each of which is naturally expressed as a STRIPS action. Starting a durative action a in state s is equivalent to applying the STRIPS action start-a in s, first verifying that prec(start-a) holds in s. Ending a in state s is equivalent to applying end-a in s , first by verifying that prec(end-a) holds in s . start-a and end-a are defined as follows:

• start-a:

• prec(start-a) = prec(a,s).

• add(start-a) = add(a, s).

• del(start-a) = del(a, s).

• end-a:

• prec(end-a) = prec(a,e).

• add(end-a) = add(e, s).

• del(end-a) = del(a, e). Example 6.2 For the durative action (stack blue green right) we have: add(end-stack green blue right) = add((stack green blue right),e) = { (on green blue), (clear green)} start-a and end-a are constrained by the duration of a, denoted d(a) and the overall precondition: end-a has to occur exactly d(a) time units after start-a, and the overall preconditions have to hold in all states between start-a and end-a. Although a has a duration, its effects apply instantaneously at the start and the end of a, respectively. The preconditions prec(a, s) and prec(a, e) are also checked instantaneously, but prec(a, o) has to hold for the entire duration of a. The structure of a durative action is summarized in Figure 6.1.

A temporal action sequence is a set of action-time pairs (a 1 , t 1 ), . . . , (a n , t n ) . Each action-time pair (a, t) is composed of a durative action a ∈ A and a scheduled start timestamp t ∈ R of a, and induces two events start-a and enda with associated timestamps t and t + d(a), respectively. Events start-a (resp. end-a) is applied in the state s t (resp. s t+d(a) ), s t (resp. s t+d(a) ) being a state timestamped with t (resp. t + d(a)). Then, the temporal transition function γ can be rewritten as: γ(s, a, t) = (γ(s t , start-a), γ(s t+d(a) , end-a)). The transition function γ(s, a, t) is defined if and only if: prec(a, s) ⊆ λ(s t ), prec(a, e) ⊆ λ(s t+d(a) ) and ∀t such that

t ≤ t ≤ t + d(a) prec(a, o) ⊆ λ(s t ).
Finally, we can define a temporal planning problem P as a formal language:

L(P) = {ω = ((a 1 , t 1 ) . . . (a n , t n ))|a i ∈ A, t i ∈ T, g ∈ G, γ(s 0 , ω, t 0 ) |= g}.
Unlike a STRIPS problem, the alphabet of the L(P) language does not only contain actions but also timestamps. However, as we will see in Section 6.3, it is possible to represent a temporal problem in the form of a STRIPS problem, so it is possible to represent a temporal problem in the form of a regular language L ST RIP S (P).

A Temporal learning problem is as follow: given a set of temporal observations Ω ⊆ L(P), is it possible to learn the temporal action model and express it into a PDDL 2.1 domain?

The key idea of our approach is to translate the temporal observations Ω into non-temporal observations Ω STRIPS , learns the DFA Σ = (S, A, γ) corresponding to the regular language L ST RIP S (P), infers a STRIPS action model from Σ and the partial and noisy observation function λ, generalizes it into PDDL planning domain and translates it into the following PDDL 2. ( a t end ( not ( holding ? x ?h ) ) ) ( a t end ( not ( c l e a r ?y ) ) ) ( a t end ( c l e a r ? x ) ) ( a t end ( handempty ?h ) ) ( a t end ( on ? x ?y ) ) ) )

Background on STRIPS Translation based Planning

Some planners (Fox and Long, 2002a;[START_REF] Halsey | Crikey-a temporal planner looking at the integration of scheduling and planning[END_REF][START_REF] Celorrio | Temporal planning with required concurrency using classical planning[END_REF][START_REF] Blanco | Forward-search temporal planning with simultaneous events[END_REF] solve temporal problems by using non-temporal planners. To that end, they convert temporal problems into classical nontemporal STRIPS problems, solve them with a non-temporal planner. Then they convert the classical plan into a temporal plan with rescheduling techniques. Temporal action models have different levels of required action concurrency [START_REF] Cushing | When is temporal planning really temporal?[END_REF]. Some of them are sequential, which means that all the plan parts containing overlapping durative actions can be rescheduled into a completely sequential succession of durative actions: each durative action starts after the previous durative action is terminated. For example, the Blocksworld domain is a sequential domain (see Figure 6.2).

One important property of sequential temporal domains is that they can be rewritten as classical domains, and therefore used by classical non-temporal planners. To solve a sequential temporal problem, we can translate each durative actions a ∈ A to a compressed STRIPS action C a that simulates all of a at once [START_REF] Coles | Managing concurrency in temporal planning using planner-scheduler interaction[END_REF]. The precondition of C a is the union of the preconditions at- start of a with the preconditions overall and at-end not achieved by the add effect at-start. The effect of C a is the effect at-start of a followed immediately by its effect at-end. Formally, the compressed action C a is defined as follows:

• prec(C a ) = prec(a, s) ∪ {{prec(a, o) ∪ prec(a, e)} \ del(a, s)} • add(C a ) = {add(a, s) \ del(a, e)} ∪ add(a, e)} • del(C a ) = {del(a, s) \ add(a, e)} ∪ del(a, e)}
Example 6. 3 For example, for Blocksworld and the action (stack green blue right), we have:

• prec(C (stack green blue right) ) = {(holding right green), (clear blue)}

• add(C (stack green blue right) ) = {(handempty right), (on green blue), (clear green)}

• del(C (stack green blue right) ) = {(holding right green), (clear blue)}

Once the durative actions are translated, the temporal problem becomes a STRIPS problem that can be solved using a classical planner. When the STRIPS problem is solved, the plan containing compressed actions is translated into a plan with durative actions executed one after another. Some temporal domains require different forms of action concurrence such as Single Hard Envelope (SHE) [START_REF] Coles | Managing concurrency in temporal planning using planner-scheduler interaction[END_REF]. SHE is a form of action concurrency where the execution of a durative action a is required for the execution of a second durative action a . Formally, a SHE is a durative action a that adds a proposition p at-start and deletes it at-end while p is an overall preconditions of a durative action a. Contrary to sequential temporal domains, for temporal domains containing SHE there exists temporal action sequences that cannot be sequentially rescheduled. Example 6. 4 For example, see the IPC Match domain1 (see Figure 6.3) and the following durative actions:

• (mend f1 m) such that (light m) ∈ prec((mend f1 m), o) • (mend f2 m) such that (light m) ∈ prec((mend f2 m), o) • (light m) such that (light m) ∈ add((light m), s) and (light m) ∈ del((light m), e)
The durative action (mend f m) cannot start before the start of the durative action (light m) and (mend f m) cannot end after the end of (light m), so (mend f m) has to start after the start of (light m) and to end before the end of (light m): it is therefore impossible to sequentially reschedule such temporal action sequences.

Generally, to solve SHE Temporal planning problems, planners start by translating durative actions into STRIPS actions. For instance, the CRICKEY planner [START_REF] Coles | Managing concurrency in temporal planning using planner-scheduler interaction[END_REF] translates each durative action a into three STRIPS actions start-a, inv-a and end-a. Then classical planners are used to solve the problem. Finally, scheduling techniques are used to translate the plans. For instance, the CRICKEY planner builds a set of partially ordered plans with the STRIPS actions. Then, a Simple Temporal Network is used to translate the set of partially ordered plans into a temporal plan.

In addition, it should be noted that there are other forms of required action concurrency besides SHE [START_REF] Cushing | When is temporal planning really temporal?[END_REF] (see Figure 6.4).

Temporal Learning

The main idea of TempAMLSI is that it is possible to learn a state machine by testing durative actions and by observing the states resulting from the

Input: Initial State Action Names Observable Predicates

Step I: Observation Generation

Step II: Observation Translation

Step III: Classical Action Model Learning

Step IV: Operators Translation executions of the durative actions and to represent it as a PDDL 2.1 planning domain. TempAMLSI assumes that it knows the names of the durative actions, i.e., the names of the transitions of the state machine, that it can test them and it is able to observe the state resulting from their applications as a set of logical propositions whose predicates are also known.

Based on these assumptions, TempAMLSI produces a set of observations Ω by using a random walk and learns as output a planning domain modeling these observations. The planning domain learned is expressed using the PDDL 2.1 language. To perform this learning, TempAMLSI translates Ω into a non temporal observation set Ω STRIPS , learns a STRIPS action model, and more precisely a set of PDDL operators ∆ STRIPS and translates it into a temporal action model, and more precisely a set of temporal PDDL 2.1 operators ∆. TempAMLSI assumes that L, A, C, S, s 0 , d are known and the observation function λ is possibly partial and noisy. No knowledge of the goal states G is required.

The TempAMLSI approach (see Figure 6.5) consists of 4 steps:

1. Observation Generation. TempAMLSI produces a set of observations Ω by using a random walk. In Section 6.4.1, we will present how TempAMLSI is able to efficiently exploit these observations by taking into account not

Observation Generation

Figure 6.6 gives an overview of this step. To generate the observations in Ω, TempAMLSI uses random walks by applying a randomly selected durative action to the initial state of the problem. If this action is feasible, the start event with the corresponding timestamp is appended to the current action sequence. Otherwise, the feasible prefix plus the infeasible start event are added to the set of negative samples I -. The procedure is repeated until the feasible prefix achieves an arbitrary size and added to the set of positive samples I + . Random walks are repeated until I + and I -achieves an arbitrary size.

Example 6. [START_REF] Grand | Action Model Learning based on Grammar Induction[END_REF] As an example, consider the following feasible sequence ω in the Blocksworld problem:

Observations Translation

In practice, the temporal sequences generated by TempAMLSI are timestamped start and end event sequences (see Example 6.5). This means that durative action (pick-up green right) starts at 0 and finishes at 1, (pick-up red left) starts at 0.5 and finishes at 1.5 and (stack green blue right) starts at 1.1 and finishes at 2.1. In the rest of this section we focus on the sample and operator translation steps. We will present two variants for these translations:

2-Operators Translation : The STRIPS action sequences contain, for each durative action a, the start action start-a and the end action end-a corresponding to the events of a durative action (see Section -6.2). This method only translates the events observed in the temporal sequences. But, it does not directly represent the overall preconditions that constrain the "life cycle" of a durative action. Indeed, for a durative action a to be executed the at-start preconditions must be checked at the start event, and the at-end preconditions must be checked at the end event, but it is also necessary that the overall preconditions are satisfied on all the duration of action a.

3-Operators Translation :

The STRIPS action sequences contain, for each durative action a, as for the 2-Operators translation, the start action start-a and the end action end-a corresponding to the events of a durative action. However, they also contains inv-a: an invariant action. This invariant action is added at each new event (a durative action starts or ends) and allows to represent the overall preconditions.

Example 6.6 Let ω be a temporal sequence such that: • 2-Operators: Each durative action a is converted into two event actions start-a and end-a. After conversion:

ω STRIPS 2OP = (start-pick-up green right), (start-pick-up red left), (end-pick-up green right), (start-stack green blue right), (end-pick-up red left), (end-stack green blue right)

• 3-Operators: In this case, each durative action a is converted into three event actions start-a, inv-a and end-a. After conversion, we have the following sequence:

ω STRIPS 3OP = (start-pick-up green right), (inv-pick-up green right), (start-pick-up red left), (inv-pick-up green right), (inv-pick-up red left), (end-pick-up green right), (inv-pick-up red left), (start-stack green blue right), (inv-pick-up red left), (inv-stack green blue right), (end-pick-up red left), (inv-stack green blue right), (end-stack green blue right)

Operators Translation

After having learned the STRIPS domain with AMLSI, TempAMLSI converts STRIPS operators into temporal operators. • prec(a,s)= prec(start-a) \ prec(end-a)

• add(a,s)=add(start-a)

• del(a,s)=del(start-a)

• prec(a,e)= prec(end-a) \ prec(start-a)

• add(a,e)=add(end-a)

• del(a,e)=del(end-a)

• prec(a,o)= prec(start-a) ∩ prec(end-a)

3-Operators at-start (resp. at-end) effects are the effects of start (resp. end) STRIPS operators. overall preconditions are the preconditions of inv STRIPS operators. And, at-start (resp. at-end) preconditions are the preconditions of start (resp. end) STRIPS operators excluding inv preconditions. Formally, 3-Operators translation is as follows:

• prec(a,s)= prec(start-a) \ prec(inv-a)

• add(a,s)=add(start-a)

• del(a,s)=del(start-a)

• prec(a,e)= prec(end-a) \ prec(inv-a)

• add(a,e)=add(end-a)

• del(a,e)=del(end-a)

• prec(a,o)= prec(inv-a)

Example 6.7 Let's go back to the Blocksworld example:

• 2-Operators: Let ∆ 2OP be the set of PDDL operators learned such that:

• prec(start-stack ?x ?y ?h) = { (holding ?x ?h) }, add(start-stack ?x ?y ?h) = ∅, del(start-stack ?x ?y ?h) = ∅

• prec(end-stack ?x ?y ?h) = { (holding ?x ?h),(clear ?y) }, add(end-stack ?x ?y ?h) = { (handempty ?h),(clear ?x),(on ?x ?y) }, del(end-stack ?x ?y ?h) = { (holding ?x ?h),(clear ?y) }

After translation, we have the following set of Temporal PDDL 2.1 operators ∆ such that:

• prec((stack ?x ?y ?h),s) = prec(start-stack ?x ?y ?h) \ prec(end-stack ?x ?y ?h) = ∅

• prec((stack ?x ?y ?h),e) = prec(end-stack ?x ?y ?h) \ prec(start-stack ?x ?y ?h) = { (clear ?y) }

• prec((stack ?x ?y ?h),o) = prec(start-stack ?x ?y ?h) ∩ prec(end-stack ?x ?y ?h) = { (holding ?x ?h) }

• add((stack ?x ?y ?h),s) = add(start-stack ?x ?y ?h) = ∅

• del((stack ?x ?y ?h),s) = del(start-stack ?x ?y ?h) = ∅

• add((stack ?x ?y ?h),e) = add(end-stack ?x ?y ?h) = { (handempty ?h),(clear ?x),(on ?x ?y) }

• del((stack ?x ?y ?h),e) = del(end-stack ?x ?y ?h) = { (holding ?x ?h),(clear ?y) }

• 3-Operators: Let ∆ 3OP be the set of PDDL operators learned such that:

• prec(start-stack ?x ?y ?h) = { (holding ?x ?h) }, add(start-stack ?x ?y ?h) = ∅, del(start-stack ?x ?y ?h) = ∅

• prec(inv-stack ?x ?y ?h) = { (holding ?x ?h) }, add(inv-stack ?x ?y ?h) = ∅, del(inv-stack ?x ?y ?h) = ∅

• prec(end-stack ?x ?y ?h) = { (holding ?x ?h),(clear ?y) }, add(end-stack ?x ?y ?h) = { (handempty ?h),(clear ?x),(on ?x ?y) }, del(end-stack ?x ?y ?h) = { (holding ?x ?h),(clear ?y) }

After translation, we have the following set of Temporal PDDL 2.1 operators ∆ such that:

• prec((stack ?x ?y ?h),s) = prec(start-stack ?x ?y ?h) \ prec(inv-stack ?x ?y ?h) = ∅

• prec((stack ?x ?y ?h),e) = prec(end-stack ?x ?y ?h) \ prec(inv-stack ?x ?y ?h) = { (clear ?y) }

• prec((stack ?x ?y ?h),o) = prec(inv-stack ?x ?y ?h) = { (holding ?x ?h) }

• add((stack ?x ?y ?h),s) = add(start-stack ?x ?y ?h) = ∅

• del((stack ?x ?y ?h),s) = del(start-stack ?x ?y ?h) = ∅

• add((stack ?x ?y ?h),e) = add(end-stack ?x ?y ?h) = { (handempty ?h),(clear ?x),(on ?x ?y) }

• del((stack ?x ?y ?h),e) = del(end-stack ?x ?y ?h) = { (holding ?x ?h),(clear ?y) }

Experiments and Evaluations

The evaluation consists in the comparison of the performance of the 2-Operators and 3-Operators variants of TempAMLSI.

Experimental Setup

Our experiments are based on 5 temporal IPC action models 2 (see Table 6.1) 3 .

More precisely we test TempAMLSI with three sequential action models (Peg Solitaire, Sokoban, Zenotravel), and two SHE action models (Match, Turn and Open). We test each IPC action model with 3 different initial states over ten runs, and we use ten randomly generated seeds for each run. Finally we generate partial observations by randomly removing a fraction of the propositions of the states, and we generate noise by changing the value of a fraction of the observable propositions. All the tests were performed on an Ubuntu 14.04 server with a multi-core Intel Xeon CPU E5-2630 clocked at 2.30 GHz with 16GB of memory. PDDL4J library [START_REF] Pellier | Pddl4j: a planning domain description library for java[END_REF] was used to generate the benchmark data.

Evaluation Metrics

TempAMLSI is evaluated using the accuracy (Zhuo et al., 2013) that measures the learned action model performance to solve new problems. Formally, Accuracy = N N * is the ratio between N, the number of correctly solved problems with the learned action model, and N * , the total number of problems to solve. In the rest of this section the accuracy is computed over 20 problems. We also report in our results the ratio of (possibly incorrectly) solved problems. A problem is incorrectly solved when a solution plan is found with the learned action model that is not correct with respect to the original action model. In the experiments, we solve the benchmark problems with the TP-SHE [START_REF] Celorrio | Temporal planning with required concurrency using classical planning[END_REF] planner. Plan validation is done with VAL, the IPC competition validation tool [START_REF] Howey | Val's progress: The automatic validation tool for pddl2. 1 used in the international planning competition[END_REF].

Discussion

Figure 6.7 shows the average performance of TempAMLSI obtained on the 5 action models of our benchmarks when varying the training data set size. The size of the training set is indicated in number of durative actions. TempAMLSI is tested with 20 experimental scenarios: the level of observability varies between 20% and 100% and the level of noise varies between 0% and 20%. We observe that 2-Operators and 3-Operators variants are accurate when the level of noise is not high (< 20%) whatever the level of observability. Only the 2-Operators variant is accurate with high level of noise. Also, we observe that the 2-Operators variant is generally more robust than the 3-Operators variant.

The fact that the 2-Operators variant is more robust than the 3-Operators variant can be explained in different ways. First of all, the fact that action sequences of 2-Operators variants are shorter than action sequences of 3-Operators variants makes DFAs easier to learn since they have fewer states. The better the DFA learning, the better the operator learning. Moreover, it is easier for 2-Operators variants than for 3-Operators variants because 2-Operators variants have less operators.

Conclusion

In this chapter we have presented TempAMLSI, a novel algorithm to learn temporal action models. TempAMLSI is built on the AMLSI approach and the idea to use classical STRIPS translation techniques: after generated a set of temporal action sequences, TempAMLSI converts temporal action sequences into non-temporal sequences. Then TempAMLSI uses AMLSI algorithm to learn a classical action model and converts it into a temporal action.

Our experimental results show that TempAMLSI is able to learn accurately both sequential and SHE temporal action models from partial and noisy datasets. However, SHE are not the only form of required action concurrency. Indeed, there exist different levels of required action concurrency for each Allen's interval algebra. So in future works, TempAMLSI could be extended to encompass more temporal relations. Also, the learned action models are STRIPScompliant, i.e. preconditions and effects are sets of logical propositions, it would be interesting to learn more complex preconditions and effects including logical quantifiers.

In the next chapter, we will present another extension of the AMSLI approach learning HTN task models.

Introduction

As we have seen it in Chapter 2, the Hierarchical Task Network (HTN) formalism [START_REF] Erol | HTN planning: Complexity and expressivity[END_REF] is very expressive and used to express a wide variety of planning problems. This formalism allows planners to exploit domain knowledge to solve problems more efficiently [START_REF] Nau | Applications of SHOP and SHOP2[END_REF] when planning problems can be naturally decomposed hierarchically in terms of tasks and task decompositions. In contrast to the classical STRIPS formalism in which only the action model needs to be specified, the HTN formalism requires to specify the task model. A task model can be primitive and compound. A primitive tasks model is described by classical actions. A compound tasks model is described using HTN methods. A HTN method describes the set of primitive and/or compound tasks requires to decompose a specific compound task. For this reason, hand-encoding HTN task models is considered more difficult and more error-prone than classical STRIPS action model. This makes it all the more necessarily to develop techniques to learn HTN task models.

In Chapter 2, we have seen that some approaches have been proposed to learn HTN task models, e.g. CAMEL [START_REF] Ilghami | Camel: Learning method preconditions for HTN planning[END_REF], HTN-Maker [START_REF] Hogg | Htn-maker: Learning htns with minimal additional knowledge engineering required[END_REF][START_REF] Hogg | Learning hierarchical task networks for nondeterministic planning domains[END_REF], LHTNDT [START_REF] Nargesian | Lhtndt: Learn htn method preconditions using decision tree[END_REF] or HTN-Learner [START_REF] Zhuo | Learning htn method preconditions and action models from partial observations[END_REF]. These approaches have several drawbacks. First, they only learn compound task models, except HTN-Learner, i.e., they consider that the primitive task model is known a priori. Also, although the majority of approaches learning only compound task models are accurate, this is not the case for methods learning both primitive and compound task models. Finally, usually these approaches take as input execution traces containing task annotations, i.e. execution traces are fully annotated with the compound tasks and their decompositions (see Figure 7.1a). Obtaining these annotation is difficult and needs a lot of human effort.

In this chapter, we present HierAMLSI (Grand et al., 2022a), an accurate learning algorithm for both compound and primitive task models robust to partial and noisy observations. HierAMLSI is built on AMLSI (see Chapter 4). Like AMLSI, HierAMLSI interacts with the environment to generate input feasible and infeasible task sequences to frame what is allowed by the targeted task model. In order to reduce the difficulty to obtain execution traces, execution traces are only partially annotated, i.e. intermediate task decomposition are unknown. Figure 7.1 gives a comparison between a fully and a partially annotated trace.

HierAMLSI contributions in HTN task model learning are fourfold:

• Output: HierAMLSI is able to learn primitive task model or compound task model or both.

• Partial and noisy observations: HierAMLSI is able to learn task models with both partial and noisy observations.

• Annotation: HierAMLSI takes as input only partially annotated execution traces that allows to reduce the difficulty to obtain annotated execution traces.

• Accuracy: HierAMLSI is accurate even with highly partial and noisy learning datasets: thus, it minimises proofreading for AI Planning experts. We show that in many HTN benchmarks HierAMLSI does not require any correction of the learned action models at all. This chapter is organized as follows. In Section 7.2 we present the problem statement. In Section 7.3, we detail the HierAMLSI steps. Then, Section 7.4 evaluates the performance of HierAMLSI on IPC benchmarks. Although HierAMLSI is able to learn the primitive task model or the compound task model or both, the primitive task model learning being done using AMLSI, this chapter focus on the compound task model learning.

Problem Statement

We propose a formal framework inspired by [START_REF] Höller | Translating totally ordered htn planning problems to classical planning problems using regular approximation of context-free languages[END_REF] in order to define the HTN learning problem. This formal framework extends the formal framework proposed in Chapter 2 to the learning problem. • L is the set of logical propositions describing the environment.

• S is the set of state labels.

• C is the set of compound tasks.

• A is the set of actions (or primitive tasks).

• M is the set of HTN method labels.

• s 0 ∈ S is the initial state.

• ω I ∈ {A ∪ C} * is the initial task network.

• G ∈ S is the set of goal state labels.

• δ is the task model.

• λ : S → 2 L is the observation function.

• σ : M → C × {A ∪ C} * is the method decomposition function 1 .

• ζ : {A ∪ C} * × S → {A ∪ C} * is the decomposition function. As for STRIPS problems, L is a set of logical propositions describing the world states, S is a set of state labels, s 0 ∈ S is the label of the initial state, G ⊆ S is the set of goal label, λ is the observation function and preconditions, positive and negative effects are given by the functions prec, add and del included in δ.

A is the set of action (or primitive task) labels and C is a set of compound (or non primitive) task labels, with C ∩ A = ∅.

Example 7.1 In the Blocksworld example, A is composed of the following primitive tasks: {(pick-up red),(pick-up blue),(pick-up green),(stack green blue),. . . } and C is composed of the following compound tasks: {(do-clear green),(do-clear blue),(do-clear red),(do-put-on green blue),. . . }.

Tasks are maintained in task networks. A task network is a sequence of tasks. Compound tasks can be decomposed by methods. The set M contains all method labels. Methods are defined by the function σ : M → C × {C A} * .

Example 7.2 As example, Figure 7.2 gives the method decomposition of the task (do-put-on green blue). There are two relevant methods, i.e. methods allowing to decompose this task: (m0-do-put-on green blue) and (m1-do-put-on green blue). The first one decomposes the task into an empty task network: σ(m0-do-put-on green blue) = ((do-put-on green blue), ∅). The second one decomposes the task into three subtasks: σ(m1-do-put-on green blue) = ((do-put-on green blue), (do-clear green),(do-clear blue),(do-move green blue) ).

A compound task c is decomposable in a state s if and only if there exists a relevant method m ∈ M such that: σ(m) = (c, ω) and prec(m) ∈ λ(s). The function ζ : {C A} * × S → {C A} * gives the decomposition function. For a totally ordered task network ω = ω 1 tω 2 , ζ is defined as follows:

ζ(ω 1 tω 2 , s) =        ω 1 tω 2 if t is a primitive task. ω 1 ω ω 2 if t is a compound task and t is decomposable in γ(ω 1 , s). ∅
Otherwise. Example 7. 3 In our example, we have:

prec(m0-do-put-on green blue) = {(on green blue), (handempty)} prec(m1-do-put-on green blue) = {(not(on green blue)), (handempty)}

Let's take our initial state s 0 , then ζ((m0-do-put-on green blue), s 0 ) = {(do-clear blue), (do-clear green), (do-move green blue)} Indeed, we have prec(m0-do-put-on green blue) ∈ λ(s 0 ) and prec(m1-do-put-on green blue)) ∈ λ(s 0 ), then ζ cannot decompose the task with the method (m0-do-put-on green blue) but can decompose the task with the method (m1-do-put-on green blue).

As ω 1 tω 2 is a totally ordered task network, ω 1 contains only primitive tasks. Indeed, as the network is totally ordered, the compound tasks are decomposed from left to right and therefore if we have to decompose t then ω 1 contains only primitive task.

We denote ω → * ω * that ω can be decomposed into ω * by 0 or more method applications. Finally, ω I is the initial task network.

Example 7. 4 In our example: ω I = {(do-put-on green blue), (do-put-on green blue)} A solution to an HTN planning problem is a task network ω with:

1. ω I → * ω, i.e. it can be reached by decomposing ω I .

2. ω ∈ A * , i.e. all tasks are primitive.

3. γ(s 0 , ω) |= g, i.e. ω is applicable in s o and results in a goal state. Finally, we can define an HTN planning problem P as a formal language:

L(P) = {ω = t 1 . . . t n |t i ∈ A, γ(s 0 , ω) |= g, ω I → * ω}.
Unlike STRIPS planning problems, the language L(P) is not necessarily regular [START_REF] Höller | Language classification of hierarchical planning problems[END_REF] and cannot be represented as a DFA. As mentioned by [START_REF] Höller | Language classification of hierarchical planning problems[END_REF][START_REF] Höller | Translating totally ordered htn planning problems to classical planning problems using regular approximation of context-free languages[END_REF] 1. L C (P) = {ω ∈ A * |γ(s 0 , ω) ∈ g}, which is defined by the state transition system defined by the preconditions and effects of the primitive tasks. This language is regular. (see Figure 4.1 in Chapter 4)

2. L H (P) = {ω ∈ A * |w I → * ω}, which is defined by the decomposition hierarchy, i.e. by the compound tasks and methods. In our example, L H (P) is defined as follows:

• ω I → (do-put-on green blue) (do-put-on red green)

• (do-put-on green blue) → ∅|(do-clear green) (do-clear blue) (move green blue) • (do-clear green) → ∅|(do-clear blue) (unstack blue green) (put-down blue)|(do-clear red) (unstack red green) (put-down red) • . . . A HTN learning problem is as follow: given a set of observations Ω ⊆ L(P), is it possible to learn the decomposition method function σ and express it into a HDDL domain?

The key idea of our approach is to learn the DFA Σ C = (S, A, γ) corresponding to the regular language L C (P), and modify the DFA by adding compound task transitions in order to encode the rules of L H (P) and approximate the language L(P)

The HierAMLSI approach

The main idea of HierAMLSI is that it is possible to learn a state machine by testing decompositions and transitions and by observing the states resulting from the executions of the decompositions and to represent it as a HDDL planning domain. HierAMLSI assumes that it knows the names of the primitive tasks, i.e., the names of the transitions of the state machine and the names of the compound tasks, that it can test and it is able to observe their decompositions and the state resulting from their applications as a set of logical propositions whose predicates are also known.

Based on these assumptions, HierAMLSI produces a set of observations Ω by using a random walk and learns as output a planning domain modeling these observations. The planning domain learned is expressed using the HDDL language. To perform this learning, HierAMLSI learns first the transition function expressed as a set of actions δ and the method decomposition function σ of a particular problem P and generalizes δ and σ as a HDDL domain ∆. HierAMLSI assumes that it knows L, A, C, S, s 0 , the decomposition function ζ is partially annotated (no knowledge about intermediate task decompositions) and the observation function λ is possibly partial and noisy. No knowledge of the goal states G is required.

The HierAMLSI approach (see Figure 7.5) consists of 4 steps:

Input: Initial State Task Names Observable Predicates

Step I: Observation Generation

Step II: DFA Learning

Step III: HTN Methods Generation

Step IV: HDDL Method Learning 1. Observation Generation. HierAMLSI produces a set of observations Ω by using a random walk. In Section 7.3.1, we will present how HierAMLSI is able to efficiently exploit these observations by taking into account not only the fact that some task are decomposable in certain states and their decomposition but also that others are not.

2. DFA Learning. HierAMLSI learns a DFA approximating the language L(P) (see Section 7.3.2).

3. HTN Methods Generation. HierAMLSI generates from the DFA learned previously a set of HTN Methods allowing to decompose all tasks observed in Ω (see Section 7.3.3).

4. HDDL Methods Learning. Once HTN Methods have been learned, HierAMLSI has to learn the HTN Methods preconditions. To do this, HierAMLSI treats HTN Methods as primitive tasks and learn an action model containing all methods using the learning and refinement techniques described in Chapter 4.

AMLSI States Machine Environnement

Randomly generated task t decomposable?

Yes/no Generation of the positive and negative task sequences

Observation Generation

The observation generation process (see Figure 7.6) is similar to the generation method described in Chapter 4. To generate the observations in Ω, HierAMLSI uses random walks by querying a State Machine. HierAMLSI chooses randomly a (primitive or compound) task t. If the task t is decomposable in a state s, HierAMLSI adds ζ(t, s), the final decomposition containing only primitive task to the current primitive task sequence. Otherwise, the feasible prefix plus the infeasible task is added to set of negative samples I -. The procedure is repeated until the feasible prefix achieves an arbitrary size and added to the set of positive samples I + . Random walks are repeated until I + and I -achieve an arbitrary size.

DFA Learning

As mentioned in Section 7.2 the language L(P) is not necessarily regular, then the purpose of this step is to learn a DFA approximating this language. More precisely, the DFA learning step is divided in 2 steps: (1) HierAMLSI learns a DFA corresponding to the language L C (P) which is defined by the state transition system defined by the preconditions and effects of the primitive tasks and ( 2) HierAMLSI adds transitions to represent compound tasks in the DFA to allow to approximate the language L(P).

Step 1: Primitive task DFA Learning AMLSI starts by using the DFA Learning algorithm described in Chapter 4 to learn the DFA containing only primitive tasks.

Step 2: Task DFA Induction Once the primitive task DFA has been learned, HierAMLSI induces the task DFA by adding compound task transitions in the DFA, i.e. by adding transitions whose labels are compound task labels.

Example 7.5 Figure 7.7 gives an example of task DFA. Suppose we have the compound task (do-put-on green blue) has been decomposed by primitive tasks {(pick-up green), (stack green blue)} in state 0 and reached the state 3. Then we add the following transitions in the DFA γ(0, (do-put-on green blue)) → 3.

HTN Methods Learning

Once the task DFA is induced HierAMLSI can directly extract HTN Methods from the task DFA. However, it is possible that a large number of HTN Methods has been generated.

Example 7.6 Let's take the task DFA in Figure 7.7. For the compound task (do-put-on green blue), HierAMLSI can generate several methods:

ω 1 = ω 2 =
(pick-up green), (stack green blue) ω 3 = (unstack blue green), (put-down blue), (pick-up green)(stack green blue) ω 4 = (do-clear green), (pick-up green)(stack green blue) ω 5 = (do-clear green), (do-clear blue), (pick-up green)(stack green blue) ω 6 = (do-clear green), (do-clear blue), (do-put-on green blue) . . . Some of these decompositions are redundant. To facilitate proof reading we want a more compact description of the HTN Methods. More precisely, we want minimizing the set of methods allowing to decompose observed compound tasks. Then, the HTN Methods learning problem can be reduce to a variant of the set cover problem [START_REF] Karp | Reducibility among combinatorial problems[END_REF] which is NP-Complete. The Greedy Approximation (GA) [START_REF] Chvátal | A greedy heuristic for the set-covering problem[END_REF]) is a classical way to approximate the solution in a polynomial time. GA is an iterative process which, at each stage, adds the method covering the largest number of decompositions. GA stops once all decompositions are covered by the method set. The main drawback of this approach is that it does not take into account the dependencies between tasks.

(do-clear green) :

ω 1 = , ω 2 = (unstack blue green), (put-down blue (do-clear blue) : ω 1 = , ω 2 = (unstack green blue), (put-down green (do-put-on green blue) : ω 1 = ω 2 = (pick-up green), (stack green blue) ω 3 = (unstack blue green), (put-down blue), (pick-up green)(stack green blue) (do-put-on blue green) : ω 1 = ω 2 = (pick-up blue), (stack blue green) ω 3 = (unstack green blue), (put-down green), (pick-up blue)(stack blue green)

(a)

Step 0: Initialization with no compound task dependency.

(do-clear green) : Example 7.7 For example, the optimal way to decompose the compound task (do-put-on green blue) is ω 1 = , ω 5 = (do-clear green), (do-clear blue), (pick-up green), (stack green blue) . So the compound task (do-put-on green blue) depends of the compound tasks (do-clear green) and (do-clear blue). So, as long as all methods for the compound tasks (do-clear green) and (do-clear blue) have been generated, GA will always prioritize ω 3 to ω 5 . Indeed, the decomposition ω 5 can be added to the current solution of GA if and only if all methods for the compound tasks (do-clear green) and (do-clear blue) have been added.

ω 1 = , ω 2 = (unstack

Heuristic Approach

We propose a sound, complete and polynomial heuristic approach taking into account dependencies between tasks. Figure 7.8 gives an example for the IPC Blocksworld domain 2 .

AMLSI starts by initializing the set of HTN methods using the decomposition function ζ observed during the observation generation step (see Section 7.3.1). For each compound task we have therefore a set of HTN Methods containing only primitive tasks and no compound task dependencies.

Example 7.8 For the compound task (do-put-on green blue) we have the three following decomposition:

ω 1 = ω 2 = (
pick-up green), (stack green blue) ω 3 = (unstack blue green), (put-down blue), (pick-up green)(stack green blue) Then, at each iteration HierAMLSI uses GA to compute a new set of HTN Methods with an additional compound task dependency. Finally, if the new HTN Method set is smaller than the one learned in the previous iteration, then it is retained.

Example 7.9 Suppose we have the two following decompositions for the compound task (do-clear blue):

ω 1 = ω 2 = (
unstack green blue), (put-down green Then, the Greedy Search return only two decompositions for the compound task (do-puton a blue):

ω 1 = ω 2 = (do-clear blue), (pick-up green)(stack green blue)
As adding a dependency reduces the number of methods required to decompose the task (do-clear blue), then these new decompositions are retained and the previous are removed.

HierAMLSI, repeats this step until it can no longer add new dependencies.

Lemma 7. 1 The Heuristic approach is sound and complete. The heuristic approach generates a set of HTN Methods M able to decompose all observed compound tasks in the observation set Ω.

Proof 7.1 During the observation generation step (see Section 7.3.1), for each generated compound task t, we have its final decomposition ζ. So at the initialization step of the Heuristic approach, there are at least one method able to decompose each observed task. The initialization is therefore sound and complete. Moreover the following steps of the Heuristic approach generates methods decomposing as many tasks as the previous steps, then the Heuristic approach is sound and complete.

Experiments and evaluations

The purpose of this evaluation is to evaluate the performance of HierAMLSI through two variants: (1) we evaluate the performance of HierAMLSI when it learns separately action or methods, and ( 2) we evaluate the performance of HierAMLSI when it learns both methods and the action model. We use several experimental scenarios: the level of noise varies between 0% and 20% and the level of observable propositions varies between 20% and 100%.

Experimental setup

Our experiments are based on 5 HDDL [START_REF] Höller | HDDL: An extension to PDDL for expressing hierarchical planning problems[END_REF][START_REF] Höller | Hierarchical planning in the IPC[END_REF] action models (see Table -7.1) from the IPC 2020 competition4 : Blocksworld, Childsnack, Transport, Zenotravel and Gripper 5 . HierAMLSI learns HTN action models from one instance. To avoid performances being biased by the initial state, HierAMLSI is evaluated with different instances. Also, for each instance, to avoid performances being biased by the generated observations, experiments are repeated ten times. All tests were performed on an Ubuntu 14.04 server with a multi-core Intel Xeon CPU E5-2630 clocked at 2.30 GHz with 16GB of memory. PDDL4J library [START_REF] Pellier | Pddl4j: a planning domain description library for java[END_REF] was used to generate the benchmark data.

Evaluation Metrics

HierAMLSI is evaluated using the accuracy (Zhuo et al., 2013) that measures the learned action model performance to solve new problems. As for the previous chapters, the accuracy is computed over 20 problems. The problems are solved with the TFD (Totally Ordered Fast Downward) planner [START_REF] Pellier | Totally and partially ordered hierarchical planners in PDDL4J library[END_REF] provided by the PDDL4J library. Plan validation is done with VAL, the IPC competition validation tool [START_REF] Howey | Val's progress: The automatic validation tool for pddl2. 1 used in the international planning competition[END_REF].

Discussion

Figure 7.9 shows the average performance of HierAMLSI obtained on the 5 action models of our benchmarks when varying the training data set size. The size of the training set is indicated in number of tasks. HierAMLSI is tested with 20 experimental scenarios: the level of observability varies between 20% and 100% and the level of noise varies between 0% and 20%. First of all, we observe that when HierAMLSI learns only the set of HTN Methods, learned action models are generally optimal (Accuracy = 100%) with 600 tasks whatever the experimental scenario. Also, 100 tasks are generally sufficient to learn accurate action models (Accuracy > 50%). Then, when HierAMLSI learns both action model and HTN Methods performances are similar when observations are noiseless. However, when observations are noisy, performances are downgraded. This is due to learning errors in the primitive task model learned. However, learned action models remain accurate when there are at least 300 tasks in the training dataset.

To conclude, we have shown experimentally that HierAMLSI learns accurate action models. More precisely, when the action model is known, HierAMLSI generally learns optimal task models. Also the performances are downgraded when AMLSI has to learn the primitive task model in addition to the set of HTN methods, but the learned task models remain accurate. Performance degradation are due to learning errors in the primitive task model.

Conclusion

In this chapter we have addressed the problem of learning HTN task models from traces with noisy and partial observations. To deal with this problem, we have we have presented HierAMLSI, an HTN extension of the AMLSI approach. This extension is composed of four steps. The first step consists in building two training sets of feasible and infeasible action sequences. In the second step, our extension induces a DFA. The third step is the generation of the HTN Methods, and the last step learns HDDL planning domain. Our experimental results show that our extension is able to learn accurately both primitive and compound task models from partial and noisy observations.

As for temporal action model learning, the learned action models are STRIPS-compliant, i.e. preconditions and effects are sets of logical propositions, it would be interesting to learn more complex preconditions and effects including logical quantifiers. Also, we have restricted ourselves to Totally Ordered HTN planning, a possible extension would be to generalize the HierAMLSI approach for Partially Ordered HTN planning.

Introduction

The field of artificial intelligence aims to design and build agents able to perceive, learn and act without any human intervention to perform complex tasks. To perform complex tasks, the agent must plan the best possible actions and execute them. To do this, the agent needs an action model. An action model is a semantic representation of the actions it can execute. In an action model, an action is represented using (1) a precondition: the set of conditions that must be satisfied to execute an action, and (2) the effects: the set of properties of the world that will be altered by executing an action. STRIPS planning is a classical method to design these action models. However, STRIPS action models are generally too restrictive to be used in real-world applications. There are other forms of action models: temporal action models allowing to represent actions that can be executed concurrently, HTN action models allowing to represent actions as tasks and subtasks, etc. These models are less restrictive, but the less restrictive the models are the more difficult they are to design. In this thesis, we are interested in approaches facilitating the acquisition of these action models based on machine learning techniques.

In this thesis, we claim that to be efficient, an action model learning approach has to be able to learn action models less restrictive than STRIPS action model. Also, learning approaches require training datasets. For these approaches to be usable in practice, the data acquisition must not be too costly. Moreover, it is imperative that the action models are sufficiently accurate to be used by planners without planning experts proofreading step. Also, the learning approach has to be robust to both partial and noisy observations, i.e. the learned action models must be accurate even if observations are partial and/or noisy. Finally, the learning approach must require few training data while avoiding the overfitting issues.

Contributions -The AMLSI Approach

In this thesis we have introduced AMLSI (Action Model Learning with State machine Interaction), a learning approach for action model acquisition.

In a first step we have shown that our approach was able to learn accurate STRIPS action models. More precisely, in Chapter 4, we presented AMLSI, a learning approach for STRIPS action models. This approach is based on the fact that STRIPS planning problems are related to state machines equivalent to regular grammars. The key idea of our approach is to learn the state machine related to the planning problem using RGI algorithms and inducing the action model from this state machine. Also, as planning problems are declared using a planning domain, our approach has to represent the STRIPS action model in the form of a PDDL planning domain. The AMLSI approach tests different actions in the environment in which the agent has to solve planning problems, observes how the environment evolves when these actions are executed, and learns the action model from its observations. We have shown experimentally that the AMLSI approach learns accurate action models even with high level of partiality and/or noise in the observations. Also, as the AMLSI approach uses both feasible and infeasible action sequences, the AMLSI approach requires few training data to learn accurate action models. Finally, we have shown that the AMLSI approach outperforms state-of-the-art approaches. Then, in Chapter 5, we have presented IncrAMLSI, an incremental extension of the AMLSI approach. This incremental extension allows us to take into account that training data acquisition beging a long term evolutive process.

In a second step we have extended the AMLSI approach to learn less restrictive action models. First of all, in Chapter 6, we have presented TempAMLSI, an accurate temporal extension of the AMLSI approach for both sequential and SHE temporal action models robust to partial and noisy observations. The key idea of the TempAMLSI approach is to use translation techniques used by some planners Fox and Long (2002a); [START_REF] Halsey | Crikey-a temporal planner looking at the integration of scheduling and planning[END_REF]; [START_REF] Celorrio | Temporal planning with required concurrency using classical planning[END_REF]; Furelos [START_REF] Blanco | Forward-search temporal planning with simultaneous events[END_REF] for the learning problem. Like AMLSI, TempAMLSI interacts with the environment to generate input feasible and infeasible action sequences to frame what is allowed by the targeted action model. Then, the temporal learning consists of three steps: (1) TempAMLSI translates temporal sequences into STRIPS sequences, (2) TempAMLSI learns a non-temporal action model with AMLSI, and then (3) translates it into a temporal action model. Then, in Chapter 7, we have presented HierAMLSI, an accurate HTN extension of the AMLSI approach for both compound and primitive task models robust to partial and noisy observations. HierAMLSI is built on AMLSI (see Chapter 4). Like AMLSI, HierAMLSI interacts with the environment to generate input feasible and infeasible action sequences to frame what is allowed by the targeted task model. Then, HierAMLSI learns the state machine, generates HTN methods and induces HTN task models.

Perspectives

Our work has many perspectives. We develop here the main ones.

AMLSI Extension

A first perspective is to extend the AMLSI approach.

The main limitation of the AMLSI approach is the expressiveness of the learned action models. Although that the AMLSI approach learns less restrictive models than STRIPS, the preconditions and effects of actions are STRIPS-compliant, i.e. preconditions and effects are sets of logical propositions. However, it is possible to have more complex preconditions and effects. First of all, the ADL Pednault (1994) formalism allows to include logical quantifier in preconditions and effects. During this thesis, we tested an extension of AMLSI for ADL action models. However, this extension was not able to learn accurate action models when the observations were noisy and/or partial. Also, the AMLSI approach does not learn any numerical features. However, some description languages and AI planning formalism allow to take into account numerical features such as probabilistic effects [START_REF] Younes | PPDDL1.0: An extension to PDDL for expressing planning domains with probabilistic effects[END_REF]; [START_REF] Sanner | Relational dynamic influence diagram language (rddl): Language description[END_REF], numerical function and fluent [START_REF] Fox | PDDL2. 1: An extension to PDDL for expressing temporal planning domains[END_REF], 2002b, 2006). A first perspective for our approach will be to extend it to respond to these limitations.

As we have stated in Chapter 2, planning problems less restrictive than STRIPS problems are more complex to solve. ML-based techniques have been proposed to facilitate the resolution of these problems such as macro actions [START_REF] Dawson | The role of preprocessing in problem solving systems[END_REF]; [START_REF] Korf | Macro-operators: A weak method for learning[END_REF] 2006) learning. The objective of these methods is to learn macro, policy or heuristics to facilitate the resolution of planning problems from execution traces. For example, macro actions are actions composed of several actions. For example, for Blocksworld, we could have the macro action pick-up-stack composed of the actions pick-up and stack. To learn these macros, the learning approaches typically take as input a set of solution plans and return the macros that have been observed often. A possible perspective could be to reuse the key idea of AMLSI to learn macros, policies and heuristics: learn the state machine related to the planning problem using grammar induction algorithms and induce, from this state machine, macros, policies and heuristics. More generally, we could extend the AMLSI approach to be able to both learn action models and learn structured knowledge in order to efficiently solve planning problems using the learned action models.

Applications

As we have seen before, the main interest of approaches learning action models is to facilitate the acquisition of these models in order to use them in real-world applications such as aerospace [START_REF] Fisher | A planning approach to monitor and control for deep space communications[END_REF]; [START_REF] Backes | Multi-mission activity planning for mars lander and rover missions[END_REF]; [START_REF] Bresina | Activity planning for the mars exploration rovers[END_REF], autonomous vehicles [START_REF] Urmson | Self-driving cars and the urban challenge[END_REF], logistics [START_REF] Cross | Dart: applying knowledge-based planning and scheduling to crisis action planning[END_REF] (2009). A natural perspective for our work would therefore be to use the AMLSI approach to facilitate the action model acquisition in these real-world applications.

A second application would be to take advantage of the interactive aspect of the AMLSI approach to facilitate the development of tools using end-user interactions. For example the programming of robots, and more precisely, the Programming by Demonstration (PbD) [START_REF] Billard | Robot programming by demonstration[END_REF] of robots. PbD is an end-user programming technique for teaching a robot new skills by demonstrating them. AMLSI can be adapted to this context to learn action models. As we have seen before, the first step of the AMLSI approach is a query phase. In the context of PbD this query phase could be the interaction between the robot and the user: the robot could ask the user about feasible and infeasible actions or tasks. In this context, the demonstrations would be the execution of the generated action sequences.

Finally, in this thesis we took advantage of the fact that planning problems are equivalent to grammars to learn action models. Also, we have seen in Chapter 3 that grammatical induction has several applications, e.g. Syntactic and Structural Pattern Recognition, Natural Language Processing Adriaans and van Zaanen ( 2004 A final application could be to use AMLSI in these application settings when a representation in the form of action models can have benefits. For example, in the field of System Behavior Modeling, grammar induction algorithms are used to discover the behavior of a system, software, industrial process etc. The behavior of these systems are represented as a grammar and can then be analyzed to automatize them [START_REF] Dupont | The QSM algorithm and its application to software behavior model induction[END_REF], detect intrusions [START_REF] Su | The essence of command injection attacks in web applications[END_REF]; [START_REF] Godefroid | Grammar-based whitebox fuzzing[END_REF] etc. In this context, AMLSI could be used to learn action models representing the behavior of these systems. The advantage would be to have a more compact and readable representation of these systems. Also, the learned action models could be directly used to end-a End event of a durative action a.

overall Time label for the whole duration of a durative action.

start-a Start event of a durative action a.

∆

A set of planning operators.

∆ P

The set of planning operators accepting L(P) and generating λ. λ Observation function.

postset(a)

The set of states where a is an ingoing transition.

preset(a) The set of states where a is an outgoing transition.

L C (P) The language defined by the state transition system.

L H (P)

The language defined by the decomposition hierarchy.

L(P)

The language accepted the set of solution plans of the planning problem P.

Ω

The training dataset.

L

The complement language of L. BS MCA (I + , I -) The border set.

I + A positive sample.

I - A negative sample.

Lat A lattice.

MCA(I + )

The Maximal Canonical Automaton.

Pr(L)

The prefix set of L.

PTA(I + ) The Prefix Tree Acceptor.

L

A language.

L(Σ)

The language accepted by Σ.

L(H(ω j ))

The language induced after reading the j first element of Ω.

L/u the right quotient of L.

List of Figures

B.1 Introduction

In this chapter, we present all the experiments performed in the framework of this thesis. For each experiment, we test each IPC action model with 3 different initial states over ten runs, and we use ten randomly generated seeds for each run. Also, we generate partial observations by randomly removing a fraction of the propositions of the states, and we generate noise by changing the value of a fraction of the observable propositions. All tests were performed on an Ubuntu 14.04 server with a multi-core Intel Xeon CPU E5-2630 clocked at 2.30 GHz with 16GB of memory. PDDL4J library [START_REF] Pellier | Pddl4j: a planning domain description library for java[END_REF] was used to generate the benchmark data.

B.2 Evaluation metrics

We evaluate AMLSI and all its extensions with three different metrics: the syntactical error [START_REF] Zhuo | Learning complex action models with quantifiers and logical implications[END_REF] that computes the distance between the original action model and the learned model, the accuracy (Zhuo et al., 2013) that expresses the capability of the learned action model to solve new problems (without proofreading). Even though the syntactical error is the most used metric in the literature, we argue that the accuracy is the most important metric in practice for planning because it measures to what extent a learned action model is useful. Indeed, it often happens that one missing precondition or effect, which amounts to a small syntactical error, makes the learned action model unable to solve planning problems. Finally, the last metric is the FScore that expresses the capability of the learned action model to generate the grammar related to the planning problem.

Formally, the syntactical error error(o) for an operator is the Hamming distance between the learned operator and the ground truth operator, i.e. the number of extra or missing predicates in the preconditions prec(o), the positive effects add(o) and the negative effects del(o) divided by the total number of possible predicates. By extension, the syntactical error for an action model composed of a set of operator O is:

E σ = 1 |O| ∑ o∈O error(o)
Then, FScore = 2.P.R P+R where R is the recall, i. Finally, the accuracy Acc = N N * is the ratio between N, the number of correctly solved problems with the learned action model, and N * , the total number of problems to solve. In the rest of this section the accuracy is computed over 20 problems. STRIPS problems are solved with Fast Downward v19.06 [START_REF] Helmert | The Fast Downward planning system[END_REF], Temporal problems are solved with the TP-SHE [START_REF] Celorrio | Temporal planning with required concurrency using classical planning[END_REF] planner and HTN problems are solved with the TFD planner [START_REF] Pellier | Totally and partially ordered hierarchical planners in PDDL4J library[END_REF] provided by the PDDL4J library. For each experiment, plan validation is done with VAL [START_REF] Howey | Val's progress: The automatic validation tool for pddl2. 1 used in the international planning competition[END_REF], which is used in the IPC competitions. In addition to the Accuracy we report the IPC score in order to compute the quality of generated plans. The score of an action model on a solved problem is the ratio between the length of a reference plan, i.e. a plan generated by the reference IPC action model, and the length of the plan generated by the learned action model. The score on an unsolved problem is 0. The score of a learned action model is the sum of its scores for all problems.

B.3 Discussion

B.3.1 STRIPS Learning

Figures B. 1 -B.48 show the average performance of AMLSI and LSONIO obtained on the 13 action models of our benchmarks when varying the training dataset size. The size of the training set is indicated in number of actions. AMLSI and LSONIO are tested with 20 experimental scenarios: the level of observability varies between 20% and 100% and the level of noise varies between 0% and 20%. We have three variants of AMLSI: (B) Base: DFA learning is done without Pairwise Sequences (PS) and without Tabu Search, (B+PS) Base + PS: DFA learning is done with PS but without Tabu Search, and (B+PS+Tabu) Base + PS + Tabu: DFA learning is done with PS and with Tabu Search during the refinement step.

Comparison with LSONIO We observe that AMLSI outperforms LSONIO whatever the size of the learning dataset in terms of accuracy or in terms of syntactical distance. We also observe that AMLSI needs very little data to obtain a relatively large accuracy (almost 90% with only a learning dataset of 200 actions) in the most difficult scenario.

Ablation study

The Base+PS variant is more robust to partial observations than the Base variant of AMLSI. This is due to the fact that DFA learned with PS are generally better that action models learned without PS. More precisely, DFA learned with PS generally have fewer states and fewer transitions. This allows for fewer false transitions which makes it easier to learn effects and preconditions. However, when observations are noisy, the Base+PS variant is not able to learn action models accurate enough to be used for planning whatever the level of observability. Only the Base+PS+Tabu variant is both robust to partial and noisy observations. Our ablation study confirms that adding unobserved Pairwise Sequences improves the learning of the DFA, and makes AMLSI more robust to partial observations while refining the preconditions and the effects by using a Tabu Search allows AMLSI to learn accurate action models with a high level of noise. shows the average performance of IncrAMLSI obtained on the 13 action models of our benchmarks when varying the convergence criterion T. IncrAMLSI is tested with 20 experimental scenarios: the level of observability varies between 20% and 100% and the level of noise varies between 0% and 20%. Whatever the experimental scenario, IncrAMLSI learns from accurate models. Also, increasing T generally leads to better results. Finally, we can observe that whatever the value of T and the experimental scenario, IncrAMLSI converges in less than 35 iterations.

Convergent Learning

Finally, we observe that the plans generated with the learned action models are generally a little longer than plansg generated with the IPC action models, even with optimal accuracy (ipc < 20). When the accuracy is not optimal, we notice that the IPC score is close to the ratio between the ipc score with optimal accuracy and the rate of solved problems, this implies that even when the accuracy is not optimal the plans are not much longer than the original plans. When observations are complete and noiseless we observe that we observe that both variants learn optimal action models. Indeed, FScore and accuracy are optimal. However, we can observe that the syntactical distance is not optimal for some action models.this is due to the fact that some at start and at end effect preconditions are encoded as overall preconditions and vice versa. Also, We observe that 2-Operators and 3-Operators variants are accurate when the level of noise is not high (< 20%) whatever the level of observability. Only the 2-Operators variant is accurate with high level of noise. Also, we observe that the 2-Operators variant is generally more robust than the 3-Operators variant. The fact that the 2-Operators variant is more robust than the 3-Operators variant can be explained in different ways. First of all, the fact that action sequences of 2-Operators variants are shorter than action sequences of 3-Operators variants makes DFAs easier to learn since they have fewer states. The better the DFA learning, the better the operator learning. Moreover, it is easier for 2-Operators variants than for 3-Operators variants because 2-Operators variants have less operators.

B.3.2 Temporal Learning

Finally, as for STRIPS learning, we observe that the plans generated with the learned action models are generally a little longer than plans generated with the IPC action models. However, we notice that the IPC score is close to the ratio between the IPC score with optimal accuracy and the rate of solved problems, this implies that even when the accuracy is not optimal the plans are not much longer than the original plans.

B.3.3 HTN Learning

The purpose of this evaluation is to evaluate the performance of HierAMLSI though two variants: (1) we evaluate the performance of HierAMLSI when only HTN Methods are learned, i.e. the action model is known and ( 2) we evaluate the performance of HierAMLSI when both HTN Methods are learned and the action model is unknown. We use several experimental scenarios: the level of noise varies between 0% and 20% and the level of observable propositions varies between 20% and 100%.

Figures B.85 -B.99 show the average performance of HierAMLSI obtained on the 5 action models of our benchmarks when varying the training data set size. The size of the training set is indicated in number of tasks. HierAMLSI is tested with 20 experimental scenarios: the level of observability varies between 20% and 100% and the level of noise varies between 0% and 20%.

First of all, we observe that when HierAMLSI learns only the set of HTN Methods, learned action models are generally optimal (Accuracy = 100%) with 600 tasks whatever the experimental scenario. Also, 100 tasks are generally sufficient to learn accurate action models (Accuracy > 50%). Then, when HierAMLSI learns both action model and HTN Methods performances are similar when observations are noiseless. However, when observations are noisy, performances are downgraded. This is due to the fact that there are learning error in the primitive task model learned. However, learned action models remain accurate when there are at least 300 tasks in the training dataset.

Finally, as for STRIPS learning, we observe that the plans generated with the learned action models are generally a little longer than plans generated with the IPC action models. However, we notice that the IPC score is close to the ratio between the IPC score with optimal accuracy and the rate of solved problems, this implies that even when the accuracy is not optimal the plans are not much longer than the original plans.

C.1 Introduction

In this chapter we give a detailed description of all the action models used in this thesis. We start by describing the STRIPS action models and more particularly their declaration in the form of PDDL planning domains. For each domain, we give a description of the domain, the list of the predicates and the list of the operators with their PDDL declaration. Then, we describe the temporal action models and more precisely their declaration in the form of PDDL 2.1 planning domains. As for PDDL domains, for each domain, we give a description of the domain, the list of the predicates and the list of the operators with their PDDL 2.1 declaration. Finally, we describe the HTN task models and more precisely their declaration in the form of HDDL planning domains. For each domain, we give a description of the domain, the list of the predicates, the list of the compound tasks and the list of the primitive tasks with their HDDL declaration and the list of HDDL methods.

C.2 STRIPS Planning Domains

C.2.1 Blocksworld

Description Classical STRIPS planning domain, where stackable blocks need to be re-assembled on a table. Using its hand, an autonomous agent can stack a block onto a block, unstack a block from a block, put down a block, or pick up a block.

Predicates

• (on ?x -block ?y -block): Thee block ?x is on the block ?y.

• (ontable ?x -block): The block ?x is on the table.

• (clear ?x -block): The block ?x is clear, i.e. there no block on ?x and it is not held .

• (holding ?x -block): The block ?x is held by the autonomous agent.

• (handempty): The autonomous agent does not hold any block. [START_REF] Fikes | STRIPS: A new approach to the application of theorem proving to problem solving[END_REF]. Les pré-conditions, les effets et les états initiaux et buts sont représentés sous la forme d'un ensemble de propositions logiques décrivant l'environnement. La modélisation STRIPS se repose sur plusieurs hypothèses restrictives :

• L'environnement est déterministe et entièrement observable. L'agent autonome connaît à tout moment l'état dans lequel il se trouve et peut donc prédire l'état dans lequel il se trouvera après l'exécution d'une action.

• Le but est spécifié en utilisant un ensemble de propriétés, c'est-à-dire des propositions logiques, que l'état final atteint par l'agent autonome doit satisfaire.

• L'environnement est statique. Seules les actions exécutées par l'agent autonome modifient l'état de l'environnent. L'agent autonome est seul et l'environnent n'a pas de dynamique interne.

• L'exécution des actions est atomique. Les actions n'ont pas de durée, les effets sont appliqués instantanément et la concurrence entre les actions n'est pas prise en compte.

• Les pré-conditions et les effets sont des ensembles de propositions logiques.

Plusieurs extensions de la modélisation STRIPS ont été proposées pour relâcher certaines hypothèses. Par exemple, les problèmes de planification temporelle [START_REF] Fox | PDDL2. 1: An extension to PDDL for expressing temporal planning domains[END_REF] permettent de modéliser des actions qui peuvent être exécutées simultanément. Cela peut être utile pour des robots à bras multiples qui peuvent manipuler plusieurs objets simultanément. Un autre exemple est celui des problèmes de planification HTN [START_REF] Erol | HTN planning: Complexity and expressivity[END_REF] qui déclarent des tâches plutôt que des actions. Il existe deux types de tâches. Les tâches primitives, similaires aux actions classiques, et les tâches complexes. Les tâches complexes sont des tâches qui, lors de la résolution du problème de planification, sont décomposées en sous-tâches (primitives et/ou complexes). Un autre exemple est celui des problèmes de planification ADL [START_REF] Pednault | ADL and the state-transition model of action[END_REF] qui permettent de modéliser des pré-conditions et des effets plus complexes avec des quantificateurs logiques, des effets conditionnels (les effets s'appliquent lorsque certaines conditions sont satisfaites), etc. Un dernier exemple est celui des problèmes de planification multi-agents [START_REF] Brenner | A multiagent planning language[END_REF] qui permettent de représenter des problèmes de planification où plusieurs agents autonomes interagissent pour atteindre un but. Cela peut être utile si nous avons plusieurs robots qui interagissent afin d'atteindre un objectif.

Plus nous relâchons des hypothèses de la modélisation STRIPS, plus les modèles d'actions sont difficiles à concevoir "à la main". D'un autre coté, plusieurs outils basés sur la planification automatique ont été développés, nous pouvons citer par exemple l'aérospatial [START_REF] Fisher | A planning approach to monitor and control for deep space communications[END_REF][START_REF] Backes | Multi-mission activity planning for mars lander and rover missions[END_REF][START_REF] Bresina | Activity planning for the mars exploration rovers[END_REF], les véhicules autonomes [START_REF] Urmson | Self-driving cars and the urban challenge[END_REF], la logistique [START_REF] Cross | Dart: applying knowledge-based planning and scheduling to crisis action planning[END_REF], la robotique [START_REF] Dvorak | A flexible ANML actor and planner in robotics[END_REF][START_REF] Lallement | HATP: Hierarchical agent-based task planner[END_REF][START_REF] Liang | iRoPro: An interactive robot programming framework[END_REF], l'industrie [START_REF] Hoffmann | Planning@sap: An application in business process management[END_REF], la cybersécurité [START_REF] Edelkamp | Early warning and intrusion detection based on combined ai methods[END_REF]. Les hypothèses de la modélisation STRIPS étant trop restrictives pour ces applications, nous avons donc besoins relâcher certaines d'entre elles ce qui rend la conception de ces modèles d'actions plus difficile. Dans cette thèse, nous nous intéressons aux outils permettant de faciliter l'acquisition de ces modèles d'actions.

D.1.2 Revue de littérature

Des outils d'ingénierie des connaissances facilitant l'écriture de modèles d'actions ont été développés. Ces outils permettent de vérifier les erreurs de syntaxe, de visualiser le modèle, etc. Cependant, ces outils nécessitent une grande expertise en planification automatique et des connaissances en génie logiciel. Des approches d'apprentissage automatique ont également été proposées pour générer automatiquement des modèles d'actions (Arora et al., 2018a;[START_REF] Celorrio | A review of machine learning for automated planning[END_REF][START_REF] Jilani | Automated knowledge engineering tools in planning: state-of-the-art and future challenges[END_REF]. Généralement, les approches d'apprentissage automatique sont utilisées pour des applications déjà existantes : soit pour automatiser des processus, comme un processus industriel par exemple, soit pour bénéficier de la flexibilité de la planification automatique, pour faciliter la maintenance de ces applications par exemple. Ces approches prennent en entrée un jeu de données d'apprentissage et apprennent un modèle d'actions. Un jeu de données d'apprentissage est une collection de données permettant d'apprendre un modèle. Dans le contexte de la planification automatique, les jeux de données d'apprentissage sont généralement des exemples d'exécutions d'agents autonomes. Ces approches sont prometteuses et certaines d'entre elles réduisent l'effort humain (Zhuo et al., 2010a) nécessaire à l'acquisition des modèles d'actions. Cependant, ces approches ne sont pas assez efficaces pour être utilisées dans des applications réelles.

Tout d'abord, la plupart de ces approches d'apprentissage apprennent des modèles d'actions STRIPS, et comme nous l'avons vu précédemment, les modèles d'actions STRIPS sont basés sur des hypothèses trop restrictives pour être utilisées dans des applications réelles. Ensuite, l'acquisition des jeux de données d'apprentissage est un processus difficile et coûteux. En effet, il faut générer les exemples, les exécuter, stocker les résultats de l'exécution. De plus, certaines approches nécessitent également d'analyser ces exemples pour en extraire une représentation symbolique. Il existe principalement deux types de jeux de données : (1) les traces de plans et (2) les marches aléatoires. La plupart des approches d'apprentissage utilisent des traces d'exécution ayant résolues un problème comme données d'apprentissage [START_REF] Yang | Learning action models from plan examples using weighted MAX-SAT[END_REF][START_REF] Aineto | Learning action models with minimal observability[END_REF][START_REF] Segura-Muros | Learning numerical action models from noisy and partially observable states by means of inductive rule learning techniques[END_REF][START_REF] Kucera | LOUGA: learning planning operators using genetic algorithms[END_REF]) et quelques approches utilisent des traces d'exécution générées aléatoirement ((Rodrigues et al., 2010a;[START_REF] Mourão | Learning STRIPS operators from noisy and incomplete observations[END_REF]).

Les traces de plans sont des traces d'exécution d'une séquence d'actions ayant accompli une tâche donnée. Par exemple, pour un véhicule autonome, une trace de plan sera la séquence d'actions permettant au véhicule de se rendre de Grenoble à Lyon. Le principal problème avec ce type de traces est qu'elles sont biaisées par la tâche à réaliser, et si nous avons trop peu de données, il y a un risque de sur-apprentissage. Par exemple, pour notre véhicule autonome, si notre jeu de données d'apprentissage ne contient que des trajets où les deux villes sont reliées par une autoroute, il est possible que le modèle d'actions appris ne puisse pas être utilisé pour des trajets reliant deux villes qui ne sont pas reliées par une autoroute. Pour limiter le risque de sur-apprentissage, nous avons donc besoin d'un grand nombre de traces d'exécution acquises à partir d'une grande variété de tâches. Cependant, comme nous l'avons mentionné précédemment, l'acquisition de ces traces est difficile et coûteuse. Les marches aléatoires sont des séquences d'actions générées aléatoirement. L'avantage de ce type de trace est qu'elle n'est pas biaisée par une tâche. En générant aléatoirement des séquences, nous pouvons couvrir un grand nombre d'exemples d'exécution tout en limitant le risque de sur-apprentissage. De plus, les marches aléatoires permettent d'acquérir des séquences d'actions infaisables, c'est-à-dire des séquences d'actions où une ou plusieurs actions ne sont pas réalisables. De plus, généralement en plus des séquences d'actions, les traces d'exécution contiennent des observations : c'est-à-dire les différents états de l'environnement observés lors de l'exécution des séquences d'actions. Ces états sont des ensembles de propositions logiques décrivant l'environnement. En pratique, obtenir des observations complètes et sans bruit est une tâche difficile et généralement les observations seront partielles et bruitées. Un état observé partiel est un état dans lequel les valeurs de certaines propositions sont inconnues. De même, un état observé bruité est un état où les valeurs de certaines propositions sont erronées. Cependant, la majorité des approches d'apprentissage ne sont capables de traiter que des observations complètes ou partielles et non bruitées [START_REF] Wang | Learning by observation and practice: An incremental approach for planning operator acquisition[END_REF][START_REF] Yang | Learning action models from plan examples using weighted MAX-SAT[END_REF][START_REF] Aineto | Learning action models with minimal observability[END_REF]. De plus, même si certaines approches ( [START_REF] Mourão | Learning STRIPS operators from noisy and incomplete observations[END_REF][START_REF] Segura-Muros | Learning numerical action models from noisy and partially observable states by means of inductive rule learning techniques[END_REF]Rodrigues et al., 2010a)

D.2 Contribution

Cette thèse contribue au domaine de la planification automatique, et plus particulièrement au domaine de l'acquisition de modèles d'actions. L'idée centrale de l'approche AMLSI est d'interagir avec l'environnement dans lequel l'agent autonome devra résoudre des problèmes de planification pour apprendre le modèle d'actions : AMSLI teste différentes actions, observe l'évolution de l'environnement lorsque ces actions sont exécutées et apprend le modèle d'actions à partir de ses observations. Cette approche est divisée en deux phases :

Phase de requêtes AMLSI teste des séquences d'actions faisables et infaisables dans l'environnement et perçoit des observations. Ces observations peuvent être partielles et/ou bruitées. L'approche AMLSI teste à la fois les séquences d'actions faisables et infaisables car les séquences d'actions infaisables permettent de minimiser le nombre d'actions (faisables) à exécuter dans l'environnement et donc de minimiser le coût d'acquisition des données d'apprentissage. De plus, les séquences d'actions seront générées de manière aléatoire afin d'éviter le problème de sur-apprentissage. Cette phase prend en entrée l'ensemble des actions que l'agent autonome peut exécuter et l'ensemble des propositions observables décrivant l'environnement. Cette phase de requêtes permet à AMLSI de construire un jeu de données d'apprentissage qui sera ensuite utilisé comme entrée de la phase d'apprentissage.

Phase d'apprentissage C'est durant cette phase qu'AMLSI apprend le modèle d'actions. Pour apprendre le modèle d'actions, AMLSI va s'appuyer sur l'induction grammatical régulière. Comme mentionné dans le chapitre 4, les problèmes de planification sont liés à des machines à états et ces machines à états sont équivalentes à des grammaires régulières. De plus, comme mentionné dans le chapitre 3, l'induction grammatical régulière est un problème bien défini [START_REF] Gold | Language identification in the limit[END_REF][START_REF] De La Higuera | Grammatical Inference: learning automata and grammars[END_REF], et de nombreux algorithmes ont été proposés pour le résoudre comme, par exemple l'algorithme RPNI [START_REF] Oncina | Inferring regular languages in polynomial updated time[END_REF]. • Apprentissage temporelle : Ensuite, nous présentons dans le chapitre 6 TempAMLSI, une approche apprenant des modèles d'actions temporels. Plusieurs planificateurs (Fox and Long, 2002a;[START_REF] Halsey | Crikey-a temporal planner looking at the integration of scheduling and planning[END_REF][START_REF] Celorrio | Temporal planning with required concurrency using classical planning[END_REF][START_REF] Blanco | Forward-search temporal planning with simultaneous events[END_REF] résolvent des problèmes de planification temporels en traduisant les problèmes temporels vers en problèmes STRIPS. TempAMLSI réutilise cette idée pour l'apprentissage de modèle : TempAMLSI apprend un modèle STRIPS en utilisant l'approche AMLSI puis traduit le modèle STRIPS en un modèle temporel. Nous montrons dans ce chapitre que TempAMLSI apprend des modèles contenant différentes formes de concurrences tel que les enveloppes [START_REF] Coles | Managing concurrency in temporal planning using planner-scheduler interaction[END_REF].

• Apprentissage HTN : Enfin, nous présentons dans le chapitre 7 HierAMLSI, une approche apprenant des modèles de tâches HTN. Comme pour TempAMLSI, HierAMLSI est basé sur l'approche AMLSI et utilise l'apprentissage STRIPS pour apprendre des modèles moins restrictifs. Nous montrons dans ce chapitre que HierAMLSI est capable d'apprendre le modèle de tâches primitives, le modèle de tâches complexes et les deux à la fois.

D.3 Perspectives

Nous allons maintenant voir les différentes perspectives possibles pour nos travaux.

D.3.1 Extension de l'approche AMLSI

Une première perspective consiste à étendre l'approche AMLSI. La principale limite de l'approche AMLSI est l'expressivité des modèles d'actions appris. Bien que l'approche AMLSI apprenne des modèles moins restrictifs que les modèles STRIPS, les pré-conditions et les effets des actions sont STRIPS, c'est-à-dire que les pré-conditions et les effets sont des ensembles de propositions logiques. Cependant, il est possible d'avoir des pré-conditions et des effets plus complexes, comme la modélisation ADL par exemple. Au cours de cette thèse, nous avons testé une extension d'AMLSI pour les modèles d'actions ADL. Cependant, cette extension n'a pas été capable d'apprendre des modèles d'actions corrects lorsque les observations étaient bruitées et/ou partielles. De plus, l'approche AMLSI n'apprend pas les aspects numériques. En effet, certaines modélisations permettent de prendre en compte des caractéristiques numériques tel que les effets probabilistes [START_REF] Younes | PPDDL1.0: An extension to PDDL for expressing planning domains with probabilistic effects[END_REF][START_REF] Sanner | Relational dynamic influence diagram language (rddl): Language description[END_REF], les fonctions numériques [START_REF] Fox | PDDL2. 1: An extension to PDDL for expressing temporal planning domains[END_REF], 2002b, 2006). Une première perspective pour notre approche serait de l'étendre pour répondre à ces limitations.

Comme mentionné dans le chapitre 2), les problèmes de planification moins restrictifs que les problèmes STRIPS sont plus complexes à résoudre. Des techniques basées sur l'apprentissage automatique ont été proposées pour faciliter la résolution de ces problèmes, comme les macro-actions : [START_REF] Dawson | The role of preprocessing in problem solving systems[END_REF][START_REF] Korf | Macro-operators: A weak method for learning[END_REF][START_REF] Botea | Macro-ff: Improving AI planning with automatically learned macro-operators[END_REF][START_REF] Castellanos-Paez | Mining useful macro-actions in planning[END_REF], la planification généralisée [START_REF] Minton | Learning search control knowledge: An explanation-based approach[END_REF][START_REF] Borrajo | Lazy incremental learning of control knowledge for efficiently obtaining quality plans[END_REF][START_REF] De La Rosa | Using cases utility for heuristic planning improvement[END_REF][START_REF] Berg | Une dernière application pourrait donc consister à utiliser AMLSI dans ces contextes applicatifs où une représentation sous forme de modèles d'actions peut présenter des avantages. Par exemple, dans le domaine de la modélisation comportementale des systèmes, des algorithmes d'induction grammatical sont utilisés pour induire le comportement d'un système, d'un logiciel, d'un processus industriel, etc[END_REF]) et l'apprentissage d'heuristique (De La Rosa et al., 2009;[START_REF] Yoon | Learning heuristic functions from relaxed plans[END_REF]. L'objectif de ces méthodes est d'apprendre des macro-actions ou des heuristiques facilitant la résolution de problèmes de planification à partir de traces d'exécution. Par exemple, les macro-actions sont des actions composées de plusieurs actions. Par exemple, pour Blocksworld, nous pourrions avoir la macro action pick-up-stack composée des actions pick-up et stack. Pour apprendre ces macros, les approches prennent généralement en entrée un ensemble de plans de solution et renvoient les macros qui ont souvent été observées. Une perspective possible serait de réutiliser l'idée centrale d'AMLSI pour apprendre ces macros et heuristiques : apprendre la machine d'état liée au problème de planification en utilisant des algorithmes d'induction de grammaire et induire à partir de cette machine d'état ces macros et heuristiques. Plus généralement, nous pourrions étendre l'approche AMLSI afin de pouvoir à la fois apprendre des modèles d'actions et apprendre à résoudre efficacement les problèmes de planification en utilisant les modèles d'actions appris.

Figure 1 . 1 :

 11 Figure 1.1: Planning Problem resolution Architecture.

Figure 1 . 2 :

 12 Figure 1.2: Human Effort for Action Model Acquisition.

Figure 1 . 3 :

 13 Figure 1.3: Overview of the AMLSI Approach.

Figure 1 . 4 :

 14 Figure 1.4: Document Organization.

Figure 2 . 1 :

 21 Figure 2.1: A first example of planning problem.

(

  : p r e d i c a t e s ( on ? xb l o c k ?yb l o c k ) ( o n t a b l e ? xb l o c k ) ( c l e a r ? xb l o c k ) ( handempty ) ( holding ? xb l o c k ) )

(

  : a c t i o n pick-up : p a r a m e t e r s ( ? xb l o c k ) : p r e c o n d i t i o n ( and ( c l e a r ? x ) ( o n t a b l e ? x ) ( handempty ) ) : e f f e c t ( and ( not ( o n t a b l e ? x ) )

  Let us now see how to describe a planning problem. For the Blocksworld example we have: ( define ( problem example ) ( :domain blocksworld ) ( : o b j e c t s red green blueb l o c k ) ( : i n i t ( c l e a r red ) ( c l e a r green ) ( c l e a r blue ) ( o n t a b l e red ) ( o n t a b l e green ) ( o n t a b l e blue )

Figure 2 . 2 :

 22 Figure 2.2: A concurrent plan to solve the Blocksworld problem.

Figure 2 . 3 :

 23 Figure 2.3: Structure of a durative action a.

  : t y p e s b l o c k hand ) ( : p r e d i c a t e s ( on ? xb l o c k ?yb l o c k ) ( o n t a b l e ? xb l o c k ) ( c l e a r ? xb l o c k ) ( handempty ?hhand ) ( holding ? xb l o c k ?hhand ) ) Finally, the domain declaration contains the declaration of all planning operators: ( : d u r a t i v ea c t i o n s t a c k : p a r a m e t e r s ( ? xb l o c k ?yb l o c k ?hhand ) : d u r a t i o n (= ? d u r a t i o n 1 ) : c o n d i t i o n ( and ( o v e r a l l ( holding ? x ?h ) ) ( a t end ( c l e a r ?y ) ) )

PDDL 2 . 1

 21 Planning Problem File Planning problems are declared in the same way as for the PDDL language: ( define ( problem example_temporal ) ( :domain blocksworld ) ( : o b j e c t s red green blueb l o c k l e f t r i g h thand ) ( : i n i t ( c l e a r red ) ( c l e a r green ) ( c l e a r blue ) ( o n t a b l e red ) ( o n t a b l e green ) ( o n t a b l e blue ) ( handempty r i g h t ) ( handempty l e f t ) ) ( : g o a l

(Figure 2 . 4 :

 24 Figure 2.4: An example of HTN method decomposition.

Figure 2 . 5 :

 25 Figure 2.5: A solution task network for the Blocksworld example.

  declare all compound and primitive tasks with their parameters: ( : t a s k do-put-on : p a r a m e t e r s ( ? xb l o c k ?yb l o c k ) ) ( : t a s k do-on-table : p a r a m e t e r s ( ? xb l o c k ) ) ( : t a s k do-move : p a r a m e t e r s ( ? xb l o c k ?yb l o c k ) ) ( : t a s k do-clear : p a r a m e t e r s ( ? xb l o c k ) ) ( : t a s k pick-up : p a r a m e t e r s ( ? xb l o c k ) ) ( : t a s k put-down : p a r a m e t e r s ( ? xb l o c k ) ) ( : t a s k s t a c k : p a r a m e t e r s ( ? xb l o c k ?yb l o c k s ) ) ( : t a s k unstack : p a r a m e t e r s ( ? xb l o c k ?yb l o c k s ) ) Then, we declare HTN methods decomposing compound tasks: ( :method m1-do-put-on : p a r a m e t e r s ( ? xb l o c k ?yb l o c k ) : t a s k ( do_put_on ? x ?y )

(

Figure 2 . 6 :

 26 Figure 2.6: Human Effort for Action Model Acquisition.

Figure 2 .

 2 Figure 2.7: Machine Learning based Approach Architecture.

Figure 2 .

 2 Figure 2.7 shows the architecture of ML-based AMA approaches. A learning algorithm takes as input a training dataset and returns an action model. This training dataset can contain a set of execution traces, structured data, human annotations and instructions, video etc. There are 3 main types of training datasets (see Figure 2.8):

Figure 2 . 8 :

 28 Figure 2.8: Effort for Training Dataset Acquisition.

  Figure 2.9: Performance Evaluation Quality.

Contents 3 . 1

 31 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Definitions and Notations . . . . . . . . . . . . . . . . . . . . . .

Property 3 . 1

 31 An RG can be represented as a Deterministic Finite Automaton (DFA) where states are non-terminal symbols and transitions are terminal symbols.

Figure 3 . 1 :

 31 Figure 3.1: A first example of automaton: Σ even accepting the even language L even .

  The minimal canonical automaton accepting an even number of b.

Figure 3 . 2 :

 32 Figure 3.2: Several automata accepting an even number of b.

Definition 3 .

 3 11 A positive sample, denoted I + , is a finite subset of sequences of the language L.

Figure 3 . 3 :

 33 Figure 3.3: The maximal canonical automaton with I + = {a, abb, bab}.

Figure 3 . 4 :Figure 3 . 5 :

 3435 Figure 3.4: The prefix tree acceptor with I + = {a, abb, bab}.

Figure 3 .

 3 4 gives the PTA({a, abb, bab}). Definition 3.16 The universal automaton of the alphabet A, denoted U A, is the automaton accepting the language A * . Example 3.10 Figure 3.5 gives the universal automaton for A = {a, b}.

Figure 3

 3 Figure 3.6: Σ even : The minimal canonical automaton accepting L even .

Figure 3 . 7 :

 37 Figure 3.7: The automata lattice with I + = {a, bab} and I -= {ababb, b, baa}.

Figure 3 .Figure 3

 33 Figure 3.8: RPNI Execution with I + = { , a, baaaba, bab, bb, bba} and I -= {ba, b, ab}.

  Figure 3.9: Deterministic Merge.

  of the negative sample. The complexity of the deterministic merge of the quotient automaton is O(|I + |) because the quotient automaton contains at most |I + | states, therefore, at most O(|I + |) merges are necessary. Then, the complexity to test the compatibility with I -is O(|I -|). Finally, we have:

  a) b bb ab (b, ba) ---(b, bb) a bba ba Table

Contents 4 . 1 4 . 4

 4144 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 STRIPS Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Observation Generation . . . . . . . . . . . . . . . . . . . 4.3.2 DFA Learning . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Operator generation . . . . . . . . . . . . . . . . . . . . . 4.3.4 Operator refinement . . . . . . . . . . . . . . . . . . . . . Formal properties . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 DFA Learning Formal Properties . . . . . . . . . . . . . . 4.4.2 Domain Learning Soundness . . . . . . . . . . . . . . . . 4.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . 4.5.2 Evaluation metrics . . . . . . . . . . . . . . . . . . . . . . 4.5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 1 :

 41 Figure 4.1: An example of DFA with pre-states and post-states.

  from Σ and the partial and noisy observation function λ and generalizes it into the following PDDL planning domain: ( : a c t i o n pick-up : p a r a m e t e r s ( ? xb l o c k ) : p r e c o n d i t i o n ( and ( c l e a r ? x ) ( o n t a b l e ? x ) ( handempty ) ) : e f f e c t ( and ( not ( o n t a b l e ? x ) )

Figure 4 . 2 :

 42 Figure 4.2: STRIPS learning process.

Figure 4 .

 4 Figure 4.3 gives an overview of this step. To generate the observations in Ω, AMLSI uses random walks by applying a randomly selected action to the initial state of the problem. If this action is feasible, it is appended to the current action sequence. Otherwise, the feasible prefix plus the infeasible action are added to the set of negative samples I -. The procedure is repeated until the feasible prefix achieves an arbitrary size and added to the set of positive samples I + . Random walks are repeated until I + and I -achieve an arbitrary size. As an example, consider below the sets of feasible sequences I + and not feasible sequences I - in the Blocksworld domain.

Figure 4 . 3 :

 43 Figure 4.3: Observation Generation Overview.

Figure 4 . 4 :

 44 Figure 4.4: Comparison of RPNI execution without and with pairwise sequences.

Proposition 4 . 1 Proof 4 . 1

 4141 AMLSI adds a polynomial number of sequences. In the worst case, |A| 2 pairwise sequences are unobserved. Then, AMLSI adds O(|A| 2 .|I + |) sequences in I -.

(

  : a c t i o n pick-up : p a r a m e t e r s ( ? xb l o c k ) : p r e c o n d i t i o n ( and ( o n t a b l e ? x ) ) : e f f e c t ( and ( holding ? x )( not ( o n t a b l e ? x ) ) )

Proposition 4 . 2

 42 Effects and preconditions refinement steps converge in a polynomial number of iterations. Proof 4.2 These two steps add effects and preconditions without removing any. In the worst case, all possible preconditions and effects are added. Then, these steps converge in O(|A|.|S|) iterations.

. 9 ) 4 . 3 Proof 4 . 3

 94343 Proposition The complexity to compute the fitness score for all candidate variants ∆ is polynomial. First of all, at each step of the Tabu Search, AMLSI tests O(|A|.|S|) candidates. Then, the complexity to compute the acceptation (resp. rejection) of positive (resp. negative) samples I + (resp. I -) is O(|I + |.|S|) (resp. O(|I -|.|S|)).Finally, the complexity to compute the number of preconditions and effects violated is O(|I + |.|S|).

Figure 4 . 5 :

 45 Figure 4.5: Refinement Steps.

Figure 4 . 6 :

 46 Figure 4.6: Average performances in terms of syntactical distance of AMLSI and LSONIO when the training dataset increases in terms of number of actions.

Figure 4 . 7 :

 47 Figure 4.7: Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions.

Figure 4 . 8 :

 48 Figure 4.8: Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions.

Figure 4 . 9 :

 49 Figure 4.9: Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of actions.

  e. the rate of sequences e accepted by the original IPC action model that are successfully accepted by the learned action model, computed as R = |{e∈E + | accept(δ,e)}| |E + | , and P is the precision, i.e. the rate of sequences e accepted by the learned action model that are also accepted by the original IPC action model, computed as P = |{e∈E + | accept(δ,e)}| |{e∈E + | accept(δ,e)}∪{e∈E -| accept(δ,e)}| . The test sets E + and E -used to compute the FScore are generated by random walks.

Figures

  Figures 4.6, 4.7, 4.8 shows the average performance of AMLSI and LSONIO obtained on the 13 action models of our benchmarks when varying the training dataset size. The size of the training set is indicated in number of actions. AMLSI

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.2 Incremental Learning . . . . . . . . . . . . . . . . . . . . . . . . 90 5.2.1 Operator Overhaul . . . . . . . . . . . . . . . . . . . . . . 91 5.2.2 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 5.3.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . 94 5.3.2 Evaluation metrics . . . . . . . . . . . . . . . . . . . . . . 95 5.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

Figure 5 . 1 :

 51 Figure 5.1: Incremental Learning Process.

Figure 5 . 2 :

 52 Figure 5.2: Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

Figure 5 .

 5 Figure 5.2 shows the average performance of IncrAMLSI obtained on the 13 domains of our benchmarks when varying the convergence criterion T. IncrAMLSI is tested with 20 experimental scenarios: the level of observability varies between 20% and 100% and the level of noise varies between 0% and 20% Whatever the experimental scenario, IncrAMLSI learns from accurate

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.3 Background on STRIPS Translation based Planning . . . . . . 102 6.4 Temporal Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 104 6.4.1 Observation Generation . . . . . . . . . . . . . . . . . . . 106 6.4.2 Observations Translation . . . . . . . . . . . . . . . . . . 106 6.4.3 Operators Translation . . . . . . . . . . . . . . . . . . . . 107 6.5 Experiments and Evaluations . . . . . . . . . . . . . . . . . . . . 110 6.5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . 110 6.5.2 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . 110 6.5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

Figure 6 . 1 :

 61 Figure 6.1: Structure of a durative action.

  Figure 6.2: Sequential Domains: An example of a concurrent plan rescheduled into a sequential plan.

Figure 6 . 3 :

 63 Figure 6.3: An example of SHE temporal domain.

  Figure 6.4: Different forms of required action concurrency.

Figure 6 . 5 :

 65 Figure 6.5: Overview of the TempAMLSI approach.

  start(pick-up green right)), (0.5, start(pick-up red left)), (1, end(pick-up green right)), (1.1, start(stack green blue right)), (1.5, end(pick-up red left)), (2.1, end(stack green blue right))

  start(pick-up green right)), (0.5, start(pick-up red left)), (1, end(pick-up green right)), (1.1, start(stack green blue right)), (1.5, end(pick-up red left)), (2.1, end(stack green blue right))

Figure 6 . 7 :

 67 Figure 6.7: Average performance of TempAMLSI when the training data set size increases in terms of number of durative actions in terms of Accuracy.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Generation . . . . . . . . . . . . . . . . . . . 121 7.3.2 DFA Learning . . . . . . . . . . . . . . . . . . . . . . . . . 121 7.3.3 HTN Methods Learning . . . . . . . . . . . . . . . . . . . 122 7.4 Experiments and evaluations . . . . . . . . . . . . . . . . . . . . 125 7.4.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . 125 7.4.2 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . 125 7.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 7.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

Figure 7 . 1 :

 71 Figure 7.1: Examples of Task Annotated Execution Traces.

  C, A, S, M, s 0 , ω I ,G, δ, λ, σ, ζ) where:

1 *Figure 7 . 2 :

 172 Figure 7.2: Method decomposition of the task (do-put-on green blue).

  Figure 7.3: A Solution Task Network.

Figure 7 .

 7 Figure 7.3 gives a solution task network for our example.Finally, we can define an HTN planning problem P as a formal language:

Figure 7 . 4 :

 74 Figure 7.4: The DFA accepting the regular language L C (P) for the Blocksworld example.

  with the DFA Σ = (S, {A ∪ C}, γ), infer the decomposition function σ: σ(m0-do-put-on b c) = ((m0-do-put-on b c), ∅) σ(m1-do-put-on b c) = ((m0-do-put-on b c), (do-clear b), (do-clear c), (do-move b c) ) . . . and the HTN methods preconditions prec(m0-do-put-on b c) = {(on b c), (handempty)} prec(m1-do-put-on b c) = {(not(on b c)), (handempty)} . . . from ζ, Σ and the partial and noisy observation function λ and express them as the following HDDL domain: ( :method m0_do_put_on : p a r a m e t e r s ( ? xb l o c k ?yb l o c k ) : t a s k ( do_put_on ? x ?y ) : p r e c o n d i t i o n ( and ( on ? x ?y ) ( handempty ) ) :ordered-subtasks ( and ) ) ( :method m1_do_put_on : p a r a m e t e r s ( ? xb l o c k ?yb l o c k ) : t a s k ( do_put_on ? x ?y ) : p r e c o n d i t i o n ( and ( handempty ) ) :ordered-subtasks ( and ( d o _ c l e a r ? x ) ( d o _ c l e a r ?y ) ( do_move ? x ?y ) ) )

Figure 7 . 5 :

 75 Figure 7.5: HTN Learning Overview.

Figure 7

 7 Figure 7.6: HTN Observation Generation Overview.

  blue green), (put-down blue (do-clear blue) : ω 1 = , ω 2 = (unstack green blue), (put-down green (do-put-on green blue) : ω 1 = , ω 2 = (do-clear green), (pick-up green), (stack green blue) (do-put-on blue green) : ω 1 = , ω 2 = (do-clear blue), (pick-up blue), (stack blue green) (b) Step 1: Induction with 1 compound task dependency.(do-clear green) : ω 1 = , ω 2 = (unstack blue green), (put-down blue (do-clear blue) : ω 1 = , ω 2 = (unstack green blue), (put-down green (do-put-on green blue) : ω 1 = , ω 2 = (do-clear green), (do-clear blue), (pick-up green), (stack green blue) (do-put-on blue green) : ω 1 = , ω 2 = (do-clear blue), (do-clear green), (pick-up blue), (stack blue green)(c) Step n: Induction with n compound task dependencies.

Figure 7 . 8 :

 78 Figure 7.8: HTN Methods Generation Example.

Lemma 7. 2

 2 The Heuristic approach is polynomial. Proof 7.2 First of all, we have O(|I + |) 3 states in the DFA. Then, in the worst case, we have a possible HTN Method for each state pair, then we have O(|I + | 2 ) possible HTN Methods in the DFA. Then, the complexity of GA is O(|I + | 3 ) in term of tested decomposition. Moreover, GA is repeated |C| 2 times. Finally, the complexity of the Heuristic approach is O(|C| 2 .|I + | 3 ).

Figure 7 . 9 :

 79 Figure 7.9: Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of Accuracy.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 8.2 Contributions -The AMLSI Approach . . . . . . . . . . . . . . 130 8.3 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 8.3.1 AMLSI Extension . . . . . . . . . . . . . . . . . . . . . . . 131 8.3.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 132

  ; Botea et al. (2005); Castellanos-Paez et al. (2018), generalized policies Minton (2012); Borrajo and Veloso (1997); de la Rosa et al. (2007, 2008) and heuristics De La Rosa et al. (2009); Yoon et al. (

  , robotics Dvorak et al. (2014); Lallement et al. (2018); Liang et al. (2022), industry Hoffmann et al. (2009), cybersecurity Edelkamp et al.

  );[START_REF] Dupont | The QSM algorithm and its application to software behavior model induction[END_REF];[START_REF] Boström | Theory-guided induction of logic programs by inference of regular languages[END_REF];[START_REF] Boström | Predicate invention and learning from positive examples only[END_REF]; Cruz-Alcázar and Vidal (1998);[START_REF] Bex | Inference of concise dtds from XML data[END_REF]; Cruz-Alcázar and Vidal (2008);[START_REF] Stein | The iupac chemical identifier -technical manual[END_REF];[START_REF] Bréhélin | Hidden markov models with patterns to learn boolean vector sequences and application to the built-in selftest for integrated circuits[END_REF];[START_REF] Raffelt | Learnlib: A library for automata learning and experimentation[END_REF];[START_REF] Berg | Regular inference for state machines with parameters[END_REF].

  Time label for the end of a durative action. at-start Time label for the start of a durative action.

3. 1 A

 1 first example of automaton: Σ even accepting the even language L even . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Several automata accepting an even number of b. . . . . . . . . . . 3.3 The maximal canonical automaton with I + = {a, abb, bab}. . . . . 3.4 The prefix tree acceptor with I + = {a, abb, bab}. . . . . . . . . . . 3.5 The universal automaton for A = {a, b}. . . . . . . . . . . . . . . . 3.6 Σ even : The minimal canonical automaton accepting L even . . . . . . 3.7 The automata lattice with I + = {a, bab} and I -= {ababb, b, baa}. 3.8 RPNI Execution with I + = { , a, baaaba, bab, bb, bba} and I -= {ba, b, ab}. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Deterministic Merge. . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 An example of DFA with pre-states and post-states. . . . . . . . . 4.2 STRIPS learning process. . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Observation Generation Overview. . . . . . . . . . . . . . . . . . . 4.4 Comparison of RPNI execution without and with pairwise sequences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Refinement Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Average performances in terms of syntactical distance of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . .

  e. the rate of sequences e accepted by the original IPC action model that are successfully accepted by the learned action model, computed as R = |{e∈E + | accept(δ,e)}| |E + | , and P is the precision, i.e. the rate of sequences e accepted by the learned action model that are also accepted by the original IPC action model, computed as P = |{e∈E + | accept(δ,e)}| |{e∈E + | accept(δ,e)}∪{e∈E -| accept(δ,e)}| . The test sets E + and E -used to compute the FScore are generated by random walks.

Figures

   shows the average performance of HierAMLSI obtained on the 5 action models of our benchmarks when varying the training data set size. The size of the training set is indicated in number of tasks. HierAMLSI is tested with 20 experimental scenarios: the level of observability varies between 20% and 100% and the level of noise varies between 0% and 20%.

Figure B. 1 :Figure B. 2 :Figure B. 3 :Figure B. 4 :Figure B. 5 :

 12345 Figure B.1: Blocksworld -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions.

  Figure B.55: Hanoi -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.56: N Puzzle -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.57: Peg Solitaire -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.58: Parking -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.59: Zenotravel -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.60: Elevator -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.61: Visit All -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.62: Logistics -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.63: Floortile -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.64: Spanner -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15.

  Figure B.65: Peg Solitaire -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of syntactical distance.

Figure D. 2 :

 2 Figure D.2: Aperçu de l'approche AMLSI

Table 2 .

 2

1: State-of-the-art of ML-based PDA approaches. From left to right: the kind of trace: Goal Oriented or Random Walk, the environment: Fully Observable, Partially Observable or Non Observable, the robustness to noise in observations, the ML techniques used, the evaluation method used: Syntactical, Semantic or Accuracy.

  3.2.1 Basic Notions of Automata Theory . . . . . . . . . . . . . 3.2.2 Regular Grammar . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Automata and Partitions . . . . . . . . . . . . . . . . . . . 3.2.3.1 Automata . . . . . . . . . . . . . . . . . . . . . . 3.2.3.2 Quotient Automata . . . . . . . . . . . . . . . .

3.2.3.3 Samples of a Language and Associated Automata . . . . . . . . . . . . . . . . . . . . . . 3.3 Regular Grammatical Induction . . . . . . . . . . . . . . . . . . 3.3.1 Identification in the limit . . . . . . . . . . . . . . . . . . 3.3.2 Search Space . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Regular Grammar Induction Algorithms . . . . . . . . . . . . . 3.4.1 Focus on the RPNI Algorithm . . . . . . . . . . . . . . . . 3.4.1.1 The RPNI Algorithm . . . . . . . . . . . . . . . 3.4.1.2 Identification in the Limit and Characteristic Sample Specification . . . . . . . . . . . . . . . 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  . In other words, a STRIPS planning

					(stack red green)		(pick-up blue)		(stack blue red)
	start	s 0	r e d ) n r e d ) ( p i c k -u p ( p u t -d o w (pick-up green) (put-down blue) ( p i c ( p k -u u t p b l -d o w u e ) n g r e e n )	s 1 s 2 s 3	(unstack red green) ( s t a c k r e d b lu e ) ( u n s t a c k r e d b lu e ) (s ta ck gr ee n re d) (u ns ta ck gr ee n re d) (s ta ck gr ee n bl ue ) (u ns ta ck gr ee n bl ue ) ( s t a c k b lu e r e d ) (stack blue green) ( u n s t a c k b lu e r e d )	s 4 s 5 s 6 s 7 s 8 s 9	(put-down blue) (pick-up green) (put-down green) (pick-up blue) (put-down blue) (pick-up red) (put-down red) (pick-up green) (put-down green) (pick-up red) (put-down red)	s 10 s 11 s 12 s 13 s 14 s 15	(unstack blue red) (stack green red) (unstack green red) (stack blue red) (unstack blue red) (stack red green) (unstack red green) (stack green blue) (unstack green blue) (stack red blue) (unstack red blue)	s 16 s 17 s 18 s 19 s 20 s 21
					(unstack blue green)
					preset(pick-up red)	postset(pick-up red)

  So, ∆ * maximizes the fitness function of the Tabu Search when observations are complete and noiseless.

Lemma 4.2 When the Tabu Search reaches the global maxima, the refinement step converges if L(Σ) ≡ L(P) and observations are complete and noiseless. Proof 4.6 When the Tabu Search reaches the global maxima, the function f (∆, I + , I -) is maximized. So, all transitions in the DFA Σ are feasible, so the Effect Refinement Step (see

Section 4.3.4)

does not add new effects. Also, all preconditions are encoded in ∆, so the Precondition Refinement Step (see

Section 4.3.4)

does not add new preconditions. So, when Tabu Search reaches the global maxima, the AMLSI refinement step converges. Theorem 4.2 If L(Σ) ≡ L(P), when observations are complete and noiseless, the refined PDDL domain ∆ learned by AMLSI is able to generate L(P) and the observations function λ.

Proof 4.7 Thanks the Lemma 4.1, we know that the global maxima of the Tabu Search is a PDDL domain able to generate L(P) and the observations function λ when L(Σ) ≡ L(P) and when observations are complete and noiseless. Also, thanks the Lemma 4.2 we know that when the Tabu Search reaches the global maxima, then the AMLSI refinement step converges. So, if L(Σ) ≡ L(P), when observations are complete and noiseless, the refined PDDL domain ∆ learned by AMLSI is able to generate L(P) and the observations function λ.

Finally, we can state the following Theorem: Theorem 4.3 Soundness. When the observation set Ω is characteristic and when observations are complete and noiseless, AMLSI learns a PDDL domain able to generate the language L(P) and the observation function λ. Proof 4.8 Thanks the Theorem 4.1 we know that AMLSI learns a DFA Σ such that L(Σ) ≡ L(P) when the observation set Ω is characteristic. Then, if L(Σ) ≡ L(P), when observations are complete and noiseless AMLSI learns a PDDL domain ∆ which is able to generate the regular language L(P) and the observation function λ.

Table 4 .

 4

	Domain Blocksworld	#Operators #Predicates E + 4 5 100 32546 49.8 33.8 E -ω + ω -
	Gripper	3	4	100 13163 51.3 33.9
	Hanoi	4	7	100 34600 50.3 33.7
	N-Puzzle	1	3	100 36626 49.9 33.7
	Peg-Solitaire	3	4	100 14508 6.9	5.3
	Parking	4	5	100 64963 50.6 34.0
	Zenotravel	5	5	100 18154 50.4 33.9
	Sokoban	2	4	100 40302 50.2 33.8
	Visit All	4	7	100 16702 50.9 35.7
	Elevator	4	6	100 13122 51.0 35.7
	Spanner	4	6	100 4628	7.0	5.1
	Logistics	6	3	100 31622 49.7 32.9
	Floortile	6	10	100 48773 51.0 37.1

1: Benchmark characteristics (from left to right): the number of operators, the number of predicates, the average size of the E + and the E - testing dataset, and the average length of the positive (resp. negative) testing sequences ω

1 .

 1 Update of the DFA with the modified version of RPNI algorithm in order to accept I t Overhaul of the PDDL operators in order to add new operators and remove preconditions and effects that are no longer compatible with I t + and I t -, and the updated DFA at iteration t (see Section 5.2.1).3.Refinement of the PDDL operators as in AMLSI to deal with noisy and partial states in I t

	Algorithm 2: Overhaul_Operator(∆ t , I t + , I t -). 1 ∆ t new ← operatorGeneration(); 2 backtrackInfeasibleEffects(∆ t );
	3 backtrackInfeasiblePreconditions(∆ t );
	4 repeat
	5	repeat
	6	backtrackPreconditions(∆ t );
	7	backtrackNegativeEffects(∆ t );
	8 9	until acceptAll(∆ t , I t + ); backtrackEffects(∆ t )
	10 until acceptAll(∆ t , I t + ); new ); 11 ∆ t+1 ← merge(∆ t , ∆ t 12 return ∆ t+1
	2.

+ and to reject I t -at iteration t (see

. + and I t -, and to produce the new domain ∆ t+1 (see

Section 4.3.4)

.

Table 5 .

 5 2 (seeChapter 4) we know that the domain returned by the refinement step is ∆ P the observation set Ω T

	Domain	#Operators #Predicates
	Blocksworld	4	5
	Gripper	3	4
	Hanoi	4	7
	N-Puzzle	1	3
	Peg-Solitaire	3	4
	Parking	4	5
	Zenotravel	5	5
	Sokoban	2	4
	Visit All	4	7
	Elevator	4	6
	Spanner	4	6
	Logistics	6	3
	Floortile	6	10

1: Benchmark characteristics.

2 -

 2 Operators at-start (resp. at-end) effects are the effects of start (resp. end) STRIPS operators. overall preconditions are the intersection of preconditions of start and end STRIPS operators. And, at-start (resp. at-end) preconditions are the preconditions of the start (resp. end) STRIPS operators excluding end (resp. start) preconditions. Formally, 2-Operators translation is as follows:

3

  |I + | denote the number of primitive tasks in the positive sample

	Domain	# Primitive Task # Compound Task # Methods # Predicates
	Blocksworld	4	4	8	5
	Gripper	3	3	4	4
	Zenotravel	4	2	5	7
	Transport	3	4	6	5
	Childsnack	6	1	2	12
			124

Table 7 .

 7 1: Benchmark domain characteristics. From left to right, the number of Primitive Tasks, the number of Compound Tasks, the number of Methods and the number of Predicates for each IPC domain.

  1.1 Planning Problem resolution Architecture. . . . . . . . . . . . . . . 1.2 Human Effort for Action Model Acquisition. . . . . . . . . . . . . 1.3 Overview of the AMLSI Approach. . . . . . . . . . . . . . . . . . . 1.4 Document Organization. . . . . . . . . . . . . . . . . . . . . . . . .

2.1 A first example of planning problem. . . . . . . . . . . . . . . . . . 2.2 A concurrent plan to solve the Blocksworld problem. . . . . . . . 2.3 Structure of a durative action a. . . . . . . . . . . . . . . . . . . . . 2.4 An example of HTN method decomposition. . . . . . . . . . . . . 2.5 A solution task network for the Blocksworld example. . . . . . . . 2.6 Human Effort for Action Model Acquisition. . . . . . . . . . . . . 2.7 Machine Learning based Approach Architecture. . . . . . . . . . . 2.8 Effort for Training Dataset Acquisition. . . . . . . . . . . . . . . . 2.9 Performance Evaluation Quality. . . . . . . . . . . . . . . . . . . .

  4.7 Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. 4.8 Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9 Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of actions. . 5.1 Incremental Learning Process. . . . . . . . . . . . . . . . . . . . . . 5.2 Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . . . . . . . . 6.1 Structure of a durative action. . . . . . . . . . . . . . . . . . . . . . 6.2 Sequential Domains: An example of a concurrent plan rescheduled into a sequential plan. . . . . . . . . . . . . . . . . . . 6.3 An example of SHE temporal domain. . . . . . . . . . . . . . . . . 6.4 Different forms of required action concurrency. . . . . . . . . . . . 6.5 Overview of the TempAMLSI approach. . . . . . . . . . . . . . . . 6.6 Observation Generation Overview. . . . . . . . . . . . . . . . . . . 6.7 Average performance of TempAMLSI when the training data set HTN Learning Overview. . . . . . . . . . . . . . . . . . . . . . . . . 7.6 HTN Observation Generation Overview. . . . . . . . . . . . . . . . 7.7 DFA Learning Step. The primitive task DFA is the DFA containing only primitive tasks, i.e. black transitions, and the task DFA contains compound tasks, i.e. dashed red transitions, in addition. 7.8 HTN Methods Generation Example. . . . . . . . . . . . . . . . . . 7.9 Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of Accuracy. . . . . . . .4 Blocksworld -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . B.5 Gripper -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . B.6 Gripper -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.7 Gripper -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.8 Gripper -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.9 Hanoi -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . .14 N Puzzle -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.15 N Puzzle -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 Peg Solitaire -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . B.20 Peg Solitaire -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . B.21 Parking -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . B.22 Parking -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.23 Parking -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.24 Parking -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 Sokoban -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . B.30 Sokoban -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.31 Sokoban -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.32 Sokoban -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of .34 Elevator -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.35 Elevator -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.36 Elevator -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.37 Visit All -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . B.38 Visit All -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.39 Visit All -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.40 Visit All -Average performances in terms of IPC score of AMLSI .44 Logistics -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.45 Floortile -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . B.46 Floortile -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.47 Floortile -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of .49 Spanner -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . B.50 Spanner -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.51 Spanner -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.52 Spanner -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.53 Blocksworld -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.54 Gripper -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.55 Hanoi -Average performance of IncrAMLSI when theconvergence criterion T varies between 1 and 15. . . . . . . . . . . B.56 N Puzzle -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.57 Peg Solitaire -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.58 Parking -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.59 Zenotravel -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.60 Elevator -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.61 Visit All -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.62 Logistics -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.63 Floortile -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.64 Spanner -Average performance of IncrAMLSI when the convergence criterion T varies between 1 and 15. . . . . . . . . . . B.65 Peg Solitaire -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of syntactical distance. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.66 Peg Solitaire -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of FScore.222 B.67 Peg Solitaire -Average performance of TempAMLSI when the .68 Peg Solitaire -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of IPC. 224 B.69 Zenotravel -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of syntactical distance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 B.70 Zenotravel -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of FScore.226 B.71 Zenotravel -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of Accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 B.72 Zenotravel -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of IPC. 228 B.73 Sokoban -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of syntactical distance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 B.74 Sokoban -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of FScore.230 B.75 Sokoban-Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of Accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 B.76 Sokoban -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of IPC. 232 B.77 Match -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of syntactical distance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 B.78 Match-Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of FScore. . . . 234 B.79 Match -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of Accuracy. . . 235 B.80 Match -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of IPC. . . . . . 236 B.81 Turn and Open -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of syntactical distance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 B.82 Turn and Open -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of FScore.238 B.83 Turn and Open -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of Accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 B.84 Turn and Open -Average performance of TempAMLSI when the training data set size increases in number of tasks in terms of IPC. 240 B.85 Blocksworld -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of FScore.241 B.86 Blocksworld -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of Accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 B.87 Blocksworld -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of IPC. 243 B.88 Gripper -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of FScore. . . . 244 B.89 Gripper -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of Accuracy. . . 245 B.90 Gripper -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of IPC. . . . . . 246 B.91 Zenotravel -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of FScore.247 B.92 Zenotravel -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of Accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 B.93 Zenotravel -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of IPC. 249 B.94 Transport -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of FScore.250 B.95 Transport -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of Accuracy.

	251
	B.96 Transport -Average performance of HierAMLSI when the
	training data set size increases in number of tasks in terms of IPC. 252
	B.97 Childsnack -Average performance of HierAMLSI when the
	training data set size increases in number of tasks in terms of FScore.253
	B.98 Childsnack -Average performance of HierAMLSI when the
	training data set size increases in number of tasks in terms of
	Accuracy. .
	B.1 Blocksworld -Average performances in terms of syntactical B.16 N Puzzle -Average performances in terms of IPC score of AMLSI
	disatnce of AMLSI and LSONIO when the training dataset and LSONIO when the training dataset increases in terms of
	increases in terms of number of actions. . . . . . . . . . . . . . . . number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . .
	B.2 Blocksworld -Average performances in terms of FScore of B.17 Peg Solitaire -Average performances in terms of syntactical
	AMLSI and LSONIO when the training dataset increases in terms disatnce of AMLSI and LSONIO when the training dataset

size increases in terms of number of durative actions in terms of Accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Examples of Task Annotated Execution Traces. . . . . . . . . . . . 7.2 Method decomposition of the task (do-put-on green blue). . . . . . . 7.3 A Solution Task Network. . . . . . . . . . . . . . . . . . . . . . . . 7.4 The DFA accepting the regular language L C (P) for the Blocksworld example. . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . B.3 Blocksworld -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . BB.10 Hanoi -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.11 Hanoi -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.12 Hanoi -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.13 N Puzzle -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . Bincreases in terms of number of actions. . . . . . . . . . . . . . . . B.18 Peg Solitaire -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . BB.25 Zenotravel -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . B.26 Zenotravel -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.27 Zenotravel -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . B.28 Zenotravel -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . Bnumber of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.33 Elevator -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . Band LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.41 Logistics -Average performances in terms of syntactical disatnce of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . B.42 Logistics -Average performances in terms of FScore of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.43 Logistics -Average performances in terms of accuracy of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . Bnumber of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . B.48 Floortile -Average performances in terms of IPC score of AMLSI and LSONIO when the training dataset increases in terms of number of actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . Btraining data set size increases in number of tasks in terms of Accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 B.99 Childsnack -Average performance of HierAMLSI when the training data set size increases in number of tasks in terms of IPC. 255 D.1 Résolution d'un problème de planification . . . . . . . . . . . . . . 308 D.2 Aperçu de l'approche AMLSI . . . . . . . . . . . . . . . . . . . . . 312

  TableB.1 shows benchmarks action model characteristics. A complete description of all action models in the IPC benchmark is given in annexes C. From left to right: the number of operators, the number of predicates, the average size of the E + and the E -testing dataset, and the average length of the positive (resp. negative) testing sequences ω + ∈ E + (resp. ω -∈ E -). From left to right, the number of Primitive Tasks, the number of Compound Tasks, the number of Methods, the number of Predicates for each IPC action model, the average size of the E + and the E -testing dataset, and the average length of the positive (resp. negative) testing sequences ω + ∈ E + (resp. ω -∈ E -).

	Domain Blocksworld	#Operators #Predicates E + 4 5 100 32546 49.8 33.8 E -ω + ω -
	Gripper	3		4	100 13163 51.3 33.9
	Hanoi		4		7	100 34600 50.3 33.7
	N-Puzzle	1		3	100 36626 49.9 33.7
	Peg-Solitaire	3		4	100 14508 6.9	5.3
	Parking	4		5	100 64963 50.6 34.0
	Zenotravel	5		5	100 18154 50.4 33.9
	Sokoban	2		4	100 40302 50.2 33.8
	Visit All	4		7	100 16702 50.9 35.7
	Elevator	4		6	100 13122 51.0 35.7
	Spanner	4		6	100 4628	7.0	5.1
	Logistics	6		3	100 31622 49.7 32.9
	Floortile	6		10	100 48773 51.0 37.1
	(a) Domain Peg Solitaire	# Operators # Predicates 1 3	Type Sequential	|E + | 100	|E -| |ω + | |ω -| 4309 3.9 3.7
	Sokoban			2	3	Sequential	100	57165 25.0 13.4
	Zenotravel		5	4	Sequential 100 22711 25.0	13.5
	Match			2	4	SHE	100	3259	4.0	4.3
	Turn and Open		5	8	SHE	100	23148 25.0 13.1
	Domain Blocksworld	# Primitive Task # Compound Task # Methods # Predicates E + 4 4 8 5 100 5804 45.5 30.1 E -ω + ω -
	Gripper		3	3		4	4		100 8336 38.4 26.5
	Zenotravel		4	2		5	7		100 3575 32.4 22.3
	Transport		3	4		6	5		100 5710 42.0 28.0
	Childsnack		6	1		2	12	100 25256 29.6 23.7
	STRIPS Learning: Our experiments are based on 13 STRIPS-Compliant IPC (c)
	benchmarks: Blocksworld, Gripper, Hanoi, N-Puzzle, Peg Solitaire, Parking,
	Zenotravel, Sokoban, Visit All, Elevator, Spanner, Logistics and Floortile. Table -
	B.1a shows our experimental setup.
	Temporal Learning: Our experiments are based on 5 temporal IPC action
	models (see Table B.1b). More precisely we test TempAMLSI with three
	Sequential action models (Peg Solitaire, Sokoban, Zenotravel), and two SHE
	action models (Match, Turn and Open).

HTN: Learning: Our experiments are based on 5 HDDL

[START_REF] Höller | HDDL: An extension to PDDL for expressing hierarchical planning problems[END_REF][START_REF] Höller | Hierarchical planning in the IPC[END_REF]

action models (see

) from the IPC 2020 competition: Blocksworld, Childsnack, Transport, Zenotravel and Gripper.

(b) From left to right: the number of operators and predicates, the temporal action model type, the average size of the E + and the E -testing dataset, and the average length of the positive (resp. negative) testing sequences ω + ∈ E + (resp. ω -∈ E -).

Table B .

 B

1: IPC Benchmark Domain Characteristics.

  Résolution d'un problème de planification être exécutées par l'agent autonome pour atteindre le but spécifié. Pour résoudre le problème de planification, l'agent autonome recherche une séquence d'actions, appelée plan, permettant d'atteindre l'état but à partir de l'état initial. La résolution des problèmes de planification est basée sur des modèles descriptifs des actions. Ils décrivent quel état ou ensemble d'états possibles peut résulter de l'exécution d'une action. Plus précisément, les modèles d'actions sont généralement utilisés pour décrire les actions. Les modèles d'actions définissent les actions en termes de pré-conditions et d'effets. Les pré-conditions expriment les propriétés de l'environnement qui doivent être satisfaites pour exécuter l'action et les effets expriment les conséquences de l'exécution de l'action sur l'environnement. En pratique, les modèles d'actions sont encodés manuellement à l'aide d'un langage déclaratif tel que le PDDL (Planning Domain Description Language)[START_REF] Ghallab | Pddl -the planning domain definition language[END_REF]. Enfin, la Figure D.1 donne l'architecture traditionnelle pour résoudre un problème de planification : un solveur, appelé planificateur, prend en entrée l'état initial, l'état but et le modèle d'actions et retourne le plan solution.La manière classique de représenter des modèles d'actions est la modélisation STRIPS (STanford Research Institute Problem Solver)

	Initial State
	Goal State	Input	Planner	Output	Plan
	Action
	Model
	Figure D.1:

  ) sont capables d'apprendre à partir d'observations partielles et bruyantes, peu d'approches sont capables de gérer des niveaux très élevés comme on peut en rencontrer dans les applications du monde réel. Enfin, les approches d'apprentissage ne sont généralement pas capables d'apprendre des modèles d'actions utilisables par les planificateurs. Une étape de relecture par un expert en planification est généralement nécessaire. Dans cette thèse, nous soutenons que, pour être efficace, une approche d'apprentissage doit aborder le triple problématique suivante : 1. Sortie: Comme nous l'avons mentionné plus haut, la planification STRIPS est trop restrictive pour les applications du monde réel. Nous devons donc proposer des approches apprenant des modèles d'actions moins restrictifs. Le modèle d'actions appris doit être correct. Un modèle d'actions est correct s'il peut être utilisé par les planificateurs pour résoudre des problèmes de planification sans nécessiter une étape de relecture. De plus, même avec des entrées ne nécessitant pas beaucoup d'efforts humains, l'acquisition de l'ensemble de données d'apprentissage est difficile et coûteuse. L'approche d'apprentissage doit nécessiter peu de données pour apprendre le modèle d'actions. Enfin, l'approche d'apprentissage doit être robuste aux observations partielles et bruitées, c'est-à-dire que l'approche d'apprentissage doit apprendre des modèles d'actions corrects même avec un niveau élevé de partialité et de bruit dans les observations. Dans cette thèse, nous nous attaquons au défi de recherche suivant : Comment apprendre automatiquement des modèles d'actions à partir d'observations partielles et bruitées ?

2. Entrée: L'acquisition de l'ensemble de données d'apprentissage doit être peu coûteuse et nécessiter peu d'efforts humains.

3. Performance:

•

  Apprentissage STRIPS : Dans un premier temps, nous montrons dans le chapitre 4 que notre approche est capable d'apprendre des modèles d'actions STRIPS corrects. L'idée centrale de notre approche est d'apprendre la machine à états reliée au problème de planification en utilisant des algorithmes d'induction grammatical régulière puis d'induire le modèle d'actions à partir de cette machine à états. De plus, comme les problèmes de planification sont déclarés à l'aide d'un domaine de planification, l'approche AMLSI représente le modèle d'actions STRIPS sous la forme d'un domaine de planification PDDL. Nous montrons expérimentalement que l'approche AMLSI apprend des modèles d'actions précis même avec un niveau élevé de partialité et/ou de bruit dans les observations. De plus, comme l'approche AMLSI utilise à la fois des séquences d'actions faisables et infaisables, elle nécessite peu de données d'apprentissage pour apprendre des modèles d'actions précis. Enfin, nous montrons que l'approche AMLSI est plus performante que les approches de l'état de l'art. Ensuite, nous présentons dans le chapitre 5 IncrAMLSI, une extension incrémentale de l'approche AMLSI. Proposer une approche incrémentale permet de répondre à deux besoins. Tout d'abord, l'acquisition de données est un processus long, les données d'apprentissage deviennent disponibles progressivement, et sont difficiles et coûteuses à obtenir. Il est donc important de pouvoir mettre à jour les modèles d'actions appris en fonction des nouvelles données entrantes sans recommencer le processus d'apprentissage depuis le début. Aussi, il est important de savoir quand arrêter l'apprentissage, c'est-à-dire avoir un critère d'arrêts, pour savoir quand arrêter le processus d'acquisition des données et donc minimiser le coût de ce processus.

• Apprentissage incrémental :

* is the Kleene operator

https://www.icaps-conference.org/competitions/

We will give a formal definition inSection 3.2.3.3

AMLSI maximizes data usability by exploiting both feasible and infeasible sequences of actions. Also, as action sequences are randomly generated, the AMLSI approach avoid the overfitting issue.

We give a complete presentation of this algorithm in Chapter 3.

https://www.icaps-conference.org/competitions/

Experimental setup are publicly available at https://github.com/maxencegrand/AMLSI

Ω t contains all observations since the beginning of the learning process.

In the previous chapter we have also used the syntactical distance and the fscore. However, we could observe that the most significant metric was the accuracy, so for this and the following chapters we will use only this metric.

An agent needs to repair fuses. To repair a fuse the agent needs a lighted match, see Annexe C for more details.

Please note that for the sake of readability the example is deliberately incomplete.

https://www.icaps-conference.org/competitions/

Experimental setup are publicly available at https://github.com/maxencegrand/AMLSI

D.3.2 Applications

Comme nous l'avons vu précédemment, le principal intérêt des approches apprenant des modèles d'actions est de faciliter l'acquisition de ces modèles afin de les utiliser dans des applications réelles tel que l'aérospatial (Fisher et al., 2000;Backes et al., 2004;Bresina et al., 2005), les véhicules autonomes (Urmson and Whittaker, 2008), la logistique (Cross and Walker, 1994), la robotique (Dvorak et al., 2014;Lallement et al., 2018;Liang et al., 2022), l'industrie (Hoffmann et al., 2009), la cybersécurité (Edelkamp et al., 2009). Une perspective directe pour notre travail serait donc d'utiliser l'approche AMLSI pour faciliter l'acquisition de ces modèles d'actions.

Une deuxième application serait de profiter de l'aspect interactif de l'approche AMLSI pour faciliter le développement d'outils utilisant des interactions avec l'utilisateur comme par exemple la programmation de robots, et plus précisément, la programmation par démonstration de robots (Billard et al., 2008). La programmation par démonstration est une technique de programmation qui permet à l'utilisateur d'enseigner de nouvelles compétences à un robot en montrant des démonstrations de ces différentes compétences. AMLSI peut être adapté à ce contexte pour apprendre des modèles d'actions. Comme nous l'avons vu précédemment, la première étape de l'approche AMLSI est une phase de requêtes. Dans le contexte de la programmation par démonstration, cette phase de requêtes pourrait être l'interaction entre le robot et l'utilisateur : le robot pourrait demander à l'utilisateur quelles sont les actions ou les tâches faisables et infaisables. Dans ce contexte, les démonstrations seraient l'exécution des séquences d'actions générées.

Enfin, dans cette thèse, nous avons tiré profit du fait que les problèmes de planification sont équivalents à des grammaires pour apprendre des modèles d'actions. De plus, nous avons vu dans le chapitre 3 que l'induction grammaticale a plusieurs applications (Adriaans and van Zaanen, 2004;Dupont et al., 2008;Boström, 1996;Boström, 1998;Cruz-Alcázar and Vidal, 1998;Bex et al., 2006;Cruz-Alcázar and Vidal, 2008;Stein et al., 2006;Bréhélin et al.,

( not ( c l e a r ? x ) )

( not ( handempty ) )

( holding ? x ) ) )

• (put-down ?x -block): The autonomous agent puts down a block on the table. ( not ( c l e a r ?y ) )

( c l e a r ? x ) ( handempty )

( on ? x ?y ) ) )

• (unstack ?x -block ?y -block): The autonomous agent unstacks a block from an other block. ( not ( on ? x ?y ) ) ) ) )

C.2.2 Gripper

Description In this domain, there are a robot called robby with grippers and a set of rooms containing balls. The goal is to transport balls from a given room to another.

Predicates

• (at-robby ?r -room): The robot robby is in the room ?r.

• (at ?b -ball ?r -room): The ball ?b is in the room ?r.

• (free ?g -gripper): The Robby's gripper ?g is free.

• (carry ?b -ball ?g -gripper): Robby is carrying the ball ?b with its gripper ?g.

Operators

• (move ?from -room ?to -room): Robby goes to the room ?to. ( not ( at-robby ? from ) ) ) )

• (pick ?b -ball ?r -room ?g -gripper): Robby picks the ball with its gripper. ( not ( a t ? b ? r ) ) 10

( not ( f r e e ?g ) ) ) )

• (drop ?b -ball ?r -room ?g -gripper): Robby drops the held ball. ( not ( c a r r y ? b ?g ) ) ) ) )

C.2.3 Hanoi

Description The Tower of Hanoi is a puzzle consisting of several rods and a number of disks of various diameters, which can slide onto any rod. It is possible to stack a disk on a second if the latter is bigger.

Predicates

• (on ?x -disk ?y -disk): The disk ?x is on the disk ?y.

• (bigger ?x -disk ?y -disk): The disk ?x is bigger than disk ?y.

• (on-location ?x -block ?l -location): The disk ?x is on the location ?l.

• (clear ?x -disk): The disk ?x is clear.

• (clear-location ?l -location): The location ?l is clear.

• (holding ?x -disk): The disk ?x is held by the autonomous agent.

• (handempty): The autonomous agent does not hold any block.

• (pick-up ?x -disk ?l -location): The autonomous agent picks a disk placed on ?l. ( not ( onlocation ? x ? l ) )

( not ( c l e a r ? x ) )

( not ( handempty ) )

( c l e a rl o c a t i o n ? l ) ( holding ? x ) ) )

• (put-down ?x -disk ?l -location): The autonomous agent puts down a disk on ?l. ( onlocation ? x ) ) )

• (stack ?x -disk ?y -disk): The autonomous agent stacks ?x on ?y. ?y must be bigger than ?x. ( not ( c l e a r ? x ) )

( not ( handempty ) )

( not ( on ? x ?y ) ) ) ) )

C.2.4 N-Puzzle

Description The N-Puzzle (also called Mystic Square) is a sliding puzzle having N 2 -1 square tiles numbered 1-(N 2 -1) on a square of N × N tiles.

Predicates

• (at ?t -tile ?p -position): The tile ?t is at ?p.

• (neighbor ?p1 -position ?p2 -position): ?p1 is neighbor of ?p2.

• (empty ?p -position): There are no tile at ?p.

Operators

• (move ?t -tile ?from -position ?to -position): The tile in position ?from is moved to the empty adjacent position ?to.

C.2.5 Peg Solitaire

Description Peg solitaire is a board game for one player involving movement of pegs on a board with holes.

Predicates

• (in-line ?x -location ?y -location ?z -location): The locations ?x, ?y and ?z are in the same line.

• (occupied ?l -location): The location ?l is occupied by a token.

• (free ?l -location): The location ?l is not occupied by any token.

• (move-ended): The move is over.

• (last-visited ?l -location): The position ?l is the last location altered (by added or removed a token) during the movement.

Operators

• (jump-new-move ?from -location ?over -location ?to -location): The player begins a new move. More precisely, The peg in the ?from location is moves into the ?to location. The peg placed in the ?over location is removed. • (jump-continue-move ?from -location ?over -location ?to -location): The player continues its move a remove the peg in the location ?over. • (end-move ?loc -location): The player ends its movement. ( not ( c l e a r ? x ) )

( not ( handempty ) )

( not ( on ? x ?y ) ) ) ) )

C.2.6 Zenotravel

Description A transportation domain involves transporting people around in planes, using different modes of movement: fast and slow.

Predicates

• (at ?p -person ?c -city): The person ?p is at the city ?c.

• (at-aircraft ?a -aircraft ?c -city): The aircraft ?a is at the city ?c.

• (in ?p -person ?a -aircraft): The person ?p is in the aircraft ?a.

• (fuel-level ?a -aircraft ?l -flevel): The fuel level ?l for the aircraft ?a.

• (next ?l1 -flevel ?l2 -flevel): If the flevel is ?l1 then the next level will be ?l2.

Operators

• (board ?p -person ?a -aircraft ?c -city): Passenger ?p boards in ?a at the city ?c. ( not ( a t ?p ? c ) ) 8 ( i n ?p ? a ) ) )

• (debark ?p -person ?a -aircraft ?c -city): Passenger ?p debarks from ?a at the city ?c. ( not ( i n ?p ? a ) ) ) )

• (fly ?a -aircraft ?c1 -city ?c2 -city ?l1 -flevel ?l2 -flevel): ?a flies from ?c1 to ?c2. Its fuel reserves are reduced from ?l1 to ?l2.

( : a c t i o n f l y : p a r a m e t e r s ( ? aa i r c r a f t ? c1 ? c2c i t y ? l 1 ? l 2f l e v e l )

: p r e c o n d i t i o n ( and ( a ta i r c r a f t ? a ? c1 ) ( f u e ll e v e l ? a ? l 1 ) ( next ? l 2 ? l 1 ) )

: e f f e c t ( and

( not ( a ta i r c r a f t ? a ? c1 ) ) ( a ta i r c r a f t ? a ? c2 )

( not ( f u e ll e v e l ? a ? l 1 ) )

( f u e ll e v e l ? a ? l 2 ) ) )

• (zoom ?a -aircraft ?c1 -city ?c2 -city ?l1 -flevel ?l2 -flevel ?l3 -flevel): ?a flies from ?c1 to ?c2. Its fuel reserves are reduced from ?l1 to ?l3. ( not ( a ta i r c r a f t ? a ? c1 ) ) ( a ta i r c r a f t ? a ? c2 )

( not ( f u e ll e v e l ? a ? l 1 ) )

( f u e ll e v e l ? a ? l 3 ) ) )

• (refuel ?a -aircraft ?c -city ?l -flevel ?l1 -flevel): ?a fills up with fuel in the city ?c. Its fuel reserves are increased from ?l to ?l1.

( : a c t i o n r e f u e l : p a r a m e t e r s ( ? aa i r c r a f t ? cc i t y ? l ? l 1f l e v e l )

: p r e c o n d i t i o n ( and

( f u e ll e v e l ? a ? l ) ( next ? l ? l 1 ) ( a ta i r c r a f t ? a ? c ) )

: e f f e c t ( and ( f u e ll e v e l ? a ? l 1 )

( not ( f u e ll e v e l ? a ? l ) ) ) ) )

C.2.7 Parking

Description The domain involves parking cars on a street with N curb locations, and where cars can be double-parked but not triple-parked.

Predicates

• (at-curb ?c -car): The car ?c is parked to a curb.

• (at-curb-num ?c -car ?c -curb): The car ?c is parked to the curb ?c.

• (behind-car ?c ?front-car -car): ?c is parked behind ?front-car.

• (car-clear ?c -car): There are no cars in the back of ?c.

• (curb-clear ?c -curb): There are no cars at ?c.

Operators

• (move-curb-to-curb ?c -car ?csrc -curb ?cdest -curb): Car parked on the curb ?csrc is moved to the curb ?cdest. ( not ( at-curb-num ? c ? c s r c ) ) ) )

• (move-curb-to-car ?c -car ?csrc -curb ?cdest -car): Car parked on the curb ?csrc is moved to the back of ?cdest. ( not ( at-curb-num ? c ? c s r c ) )

( not ( at-curb ? c ) ) ) )

• (moce-car-to-curb ?c -car ?csrc -car ?cdest -curb): Car parked on the back of ?csrc is moved to the curb ?cdest. ( not ( behind-car ? c ? c s r c ) )

( at-curb ? c ) ) )

• (move-car-to-car ?c -car ?csrc -car ?cdest -car): Car parked on the back of ?csrc is moved to the back of ?cdest.

C.2.8 Sokoban

Description Sokoban is a puzzle in which the player pushes boxes around in a warehouse, trying to get them to storage locations

Predicates

• (at ?o -box ?l -location): the box ?o is at ?l.

• (at-robot ?l -location): The robot is at ?l.

• (adjacent ?l1 -location ?l2 -location ?d -direction): The robot can move from ?l1 to ?l2, and ?l2 is on the ?d of ?l1.

• (clear ?l -location): There are nothing at ?l.

Operators

• (move ?from -location ?to -location ?dir -direction): The player moves to the ?dir of the location ?from to the location ?to. ( not ( at-robot ? from ) ) ) )

• (push ?rloc -location ?bloc -location ?floc -location ?dir -direction ?b -box):

The player pushes the box ?b to the location ?floc. The initial position of the player was ?rloc and its final position is the initial position of the box: ?bloc.

C.2.9 Visit All

Description An agent in the middle of a square grid n × n must visit all the cells in the grid

Predicates

• (connected ?x -place ?y -place): Places ?x and ?y are connected.

• (at-robot ?x -place):

The robot is at ?x.

• (visited ?x -place): The robot has already visited ?x.

Operators

• (move ?curpos -place ?nextpos -place): Agent moves from the ?curpos to the ?nextpos. ( not ( at-robot ? curpos ) ) ) )

• (visit ?pos -place): Agent marks its position as visited.

C.2.10 Elevator

Description Transport a number of passengers with an elevator from their origin to their destination floors. With explicit control over the passengers that get in or out of the lift.

Predicates

• (origin ?p -passenger ?f -floor): ?p waits for the elevator at the floor ?f.

• (destin ?p -passenger ?f -floor): ?p wants to go to the floor ?f.

• (ontable ?x -block): The block ?x is on the table.

• (above ?f1 -floor ?f2 -floor): ?f1 is above ?f2.

• (boarded ?p -passenger): ?p is on board of the elevator.

• (served ?p -passenger): ?p has been served by the elevator.

• (lift-at ?f -floor):

The elevator is at floor ?f.

Operators

• (board ?f -floor ?p -passenger): Passenger ?p boards the elevator at floor ?f. ( not ( boarded ?p ) )

7

( not ( served ?p ) ) )

8

: e f f e c t ( boarded ?p ) )

• (depart ?f -floor ?p -passenger): Passenger ?p exits the elevator at floor ?f. • (up ?f1 -floor ?f2 -floor): The elevator goes up to floor ?f2 from floor ?f1. ( not ( l i f ta t ? f 1 ) ) ) )

• (down ?f1 -floor ?f2 -floor): The elevator goes down to floor ?f2 from floor ?f1. ( l i f ta t ? f 1 )

5

( above ? f 2 ? f 1 ) )

6

: e f f e c t ( and

C.2.11 Floortile

Description A set of robots use different colors to paint patterns in floor tiles.

The robots can move around the floor tiles in four directions (up, down, left, and right). Robots paint with one color at a time but can change their spray guns to any available color. However, robots can only paint the tile that is in front (up) and behind (down) them, and once a tile has been painted, no robot can stand on it.

Predicates

• (robot-at ?r -robot ?x -tile): Robot ?r is at tile ?t.

• (robot-has ?r -robot ?c -color): Robot ?r has the color ?c ?t.

• (above ?x -tile ?y -tile): ?y is above ?x.

• (below ?x -tile ?y -tile): ?y is below ?x.

• (rightOf ?x -tile ?y -tile): ?y is to the right of ?x.

• (leftOf ?x -tile ?y -tile): ?y is to the left of ?x.

• (clear ?x -tile): The tile ?x is clear, i.e. there no robot or paint on ?x.

• (painted ?x -tile ?c -color): ?x is painted in ?c.

• (available-color ?c -color): ?c is available.

• (free-color ?r -robot): Robot ?x has no color.

Operators

• (change-color ?r -robot ?c -color ?c2 -color): Robot ?r replaces the color ?c with the color ?c2. ( robot-has ? r ? c2 ) ) )

• (paint-up ?r -robot ?y -tile ?x -tile ?c -color): Robot ?r, place on tile ?x, paint the tile above it (?y) with color ?c. ( p a i n t e d ?y ? c ) ) )

• (paint-down ?r -robot ?y -tile ?x -tile ?c -color): Robot ?r, place on tile ?x, paint the tile below it (?y) with color ?c. ( p a i n t e d ?y ? c ) ) )

• (up ?r -robot ?x -tile ?y -tile): Robot ?r goes on the tile above. ( not ( robot-at ? r ? x ) )

( c l e a r ? x )

( not ( c l e a r ?y ) ) ) )

• (down ?r -robot ?x -tile ?y -tile): Robot ?r goes on the tile below. ( not ( robot-at ? r ? x ) )

( c l e a r ? x )

( not ( c l e a r ?y ) ) ) )

• (right ?r -robot ?x -tile ?y -tile): Robot ?r goes on the tile to its right. ( not ( robot-at ? r ? x ) )

( c l e a r ? x )

( not ( c l e a r ?y ) ) ) )

• (left ?r -robot ?x -tile ?y -tile): Robot ?r goes on the tile to its left.

C.2.12 Logistics

Description Transport packages within cities via trucks, and between cities via airplanes. Locations within a city are directly connected (trucks can move between any two such locations), and so are the cities.

Predicates

• in-city ?loc -place ?c -city): The place ?loc is in ?c.

• (at ?obj -physobj ?loc -place): The physical object ?obj is at place ?loc.

• (in ?pkg -package ?veh -vehicle): The package ?pkg is in the vehicle ?veh.

Operators

• (load-truck ?pkg -package ?t -truck ?loc -place): The package ?pkg is loaded in the truck ?t at the place ?loc. ( not ( a t ?pkg ? l o c ) ) 10 ( i n ?pkg ? t ) ) )

• (unload-truck ?pkg -package ?t -truck ?loc -place): The package ?pkg is unloaded from the truck ?t at the place ?loc. ( not ( i n ?pkg ? t ) ) ( a t ?pkg ? l o c ) ) )

• (load-airplane ?pkg -package ?a -airplane ?loc -place): The package ?pkg is loaded in the airplane ?a at the place ?loc. • (unload-airplane ?pkg -package ?a -airplane ?loc -place): The package ?pkg is unloaded from the airplane ?a at the place ?loc. • (fly-airplane ?a -airplane ?loc-from -airport ?loc-to -airport): The airplane flies from the airport ?loc-from to the airport ?loc-to. ( not ( a t ? a ? loc-from ) )

7 ( a t ? a ? loc-to ) ) ) )

C.2.13 Spanner

Description A worker is in a shed, containing a number of spanners, and at a gate some distance away there are a number of nuts that must be tightened. The spanners are fragile and can be used only once to tighten a nut.

Predicates

• (at ?m -locatable ?l -location): ?m is at ?l.

• (carrying ?m -man ?s -spanner): The man ?m carries the spanner ?s.

• (useable ?s -spanner): There are no man carrying ?s.

• (link ?l1 -location ?l2 -location): There are a link between ?l1 and ?l2.

• (tightened ?n -nut): The nut ?n is tightened.

• (loose ?n -nut): The nut ?n is loosed.

Operators

• (walk ?start -location ?end -location ?m -man): The worker ?m walks from ?start to ?end. ( not ( a t ?m ? s t a r t ) ) ( a t ?m ? end ) ) )

• (pickup-spanner ?l -location ?s -spanner ?m -man): ?m picks the spanner ?s at ?l. ( not ( a t ? s ? l ) )

( c a r r y i n g ?m ? s ) ) )

• (tighten-nut ?l -location ?s -spanner ?m -man ?n -nut): ?m tightens the nut with its spanner.

Predicates

• (handfree): The hand is free.

• (unused ?m -match): Match ?m is unused.

• (mended ?f -fuse): Fuse ?f is mended.

• (light ?m -match): Match ?m is light.

Operators

• (light-match ?m -match): The agent lights the match ?m. 7 ( a t s t a r t ( l i g h t ?m) ) 8 ( a t end ( not ( l i g h t ?m) ) ) ) )

• (mend-fuse ?f -fuse ?m -match): The agent repairs the fuse ?f by lighting himself with the match ?m.

C.3.2 Turn and Open

Description In this domain, there are several robots with grippers and a set of rooms containing balls. The goal is to transport balls from a given room to another. There are doors that must be open to move from one room to another.

In order to open a given door, the robot must turn the doorknob and open the door at the same time.

Predicates

• (at-robby ?r -robot ?x -room): Robot ?r is at room ?x.

• (at ?o -obj ?x -room): Object ?o is at room ,x.

• (free ?r -robot ?g -gripper): Robot ?r's gripper ?g is free.

• (carry ?r -robot ?o -obj ?g -gripper): Robot ?r is carrying ?o with its gripper ?g.

• (connected ?x -room ?y -room ?d -door): Rooms ?x and ?y are connected with the door ?d.

• (open ?d -door): ?d is opened.

• (closed ?d -door): ?d is closes.

• (doorknob-turned ?d -door ?g -gripper): The door's doorknob is turned by ?g.

Operators

• (turn-doorknob ?r -robot ?from ?to -room ?d -door ?g -gripper): Robot turns the doorknob of the door. • (open-door)?r -robot ?from ?to -room ?d -door ?g -gripper: Robot opens the door. : e f f e c t ( and ( a t end ( at-robby ? r ? t o ) ) ( a t s t a r t ( not ( at-robby ? r ? from ) ) ) ) )

• (pick ?r -robot ?obj -obj ?room -room ?g -gripper): Robot picks the ball with its gripper. ( a t s t a r t ( at-robby ? r ?room ) ) ( a t s t a r t ( f r e e ? r ?g ) ) ) : e f f e c t ( and 12 ( a t end ( c a r r y ? r ? o b j ?g ) ) 13 ( a t s t a r t ( not ( a t ? o b j ?room ) ) ) 14 ( a t s t a r t ( not ( f r e e ? r ?g ) ) ) ) )

• (drop ?r -robot ?obj -obj ?room -room ?g -gripper): Robot drops the ball holds by its gripper. 9 ( a t s t a r t ( at-robby ? r ?room ) ) 10 ( a t s t a r t ( f r e e ? r ?g ) ) )

11 : e f f e c t ( and 12 ( a t end ( c a r r y ? r ? o b j ?g ) ) 13 ( a t s t a r t ( not ( a t ? o b j ?room ) ) ) 14 ( a t s t a r t ( not ( f r e e ? r ?g ) ) ) ) )

C.3.3 Peg Solitaire

Description Peg solitaire is a board game for one player involving movement of pegs on a board with holes.

Predicates

• (in-line ?x -location ?y -location ?z -location): The locations ?x, ?y and ?z are in the same line.

• (occupied ?l -location): The location ?l is occupied by a token.

• (free ?l -location): The location ?l is not occupied by any token.

Operators

• (jump ?from -location ?over -location ?to -location): The player jumps from ?from to ?to. More precisely, The peg in the ?from location is moves into the ?to location. The peg placed in the ?over location is removed.

C.3.4 Zenotravel

A transportation domain involves transporting people around in planes, using different modes of movement: fast and slow.

Description

Predicates

• (at ?x -locatable ?c -city): The locatable ?x is at the city ?c.

• (in ?p -person ?a -aircraft): The person ?p is in the aircraft ?a.

• (fuel-level ?a -aircraft ?l -flevel): The fuel level ?l for the aircraft ?a.

• (next ?l1 -flevel ?l2 -flevel): If the flevel is ?l1 then the next level will be ?l2.

Operators

• (board ?p -person ?a -aircraft ?c -city): Passenger ?p boards in ?a at the city ?c. : e f f e c t ( and ( a t s t a r t ( not ( a t ? a ? c1 ) ) ) ( a t end ( a t ? a ? c2 ) ) ( a t s t a r t ( not ( f u e ll e v e l ? a ? l 1 ) ) ) ( a t end ( f u e ll e v e l ? a ? l 2 ) ) ) )

• (zoom ?a -aircraft ?c1 -city ?c2 -city ?l1 -flevel ?l2 -flevel ?l3 -flevel): ?a flies from ?c1 to ?c2. Its fuel reserves are reduced from ?l1 to ?l3. ( a t s t a r t ( a t ? a ? c1 ) ) ( a t s t a r t ( f u e ll e v e l ? a ? l 1 ) ) ( a t s t a r t ( next ? l 2 ? l 1 ) ) ( a t s t a r t ( next ? l 3 ? l 2 ) ) ) : e f f e c t ( and ( a t s t a r t ( not ( a t ? a ? c1 ) ) ) ( a t end ( a t ? a ? c2 ) ) ( a t end ( not ( f u e ll e v e l ? a ? l 1 ) ) ) 285 ( a t end ( f u e ll e v e l ? a ? l 3 ) ) ) )

• (refuel ?a -aircraft ?c -city ?l -flevel ?l1 -flevel): ?a fills up with fuel in the city ?c. Its fuel reserves are increased from ?l to ?l1. 12 ( a t s t a r t ( not ( f u e ll e v e l ? a ? l ) ) ) ) )

C.3.5 Sokoban

Description Sokoban is a puzzle in which the player pushes stones around in a warehouse, trying to get them to storage locations

Predicates

• (at ?t -thing ?l -location): The thing ?l is at ?l.

• (adjacent ?l1 -location ?l2 -location ?d -direction): The robot can move from ?l1 to ?l2, and ?l2 is on the ?d of ?l1.

• (clear ?l -location): There are nothing at ?l.

Operators

• (move ?p -player ?from -location ?to -location ?dir -direction): The player moves to the ?dir of the location ?from to the location ?to. ( a t end ( c l e a r ? from ) ) ) )

• (push ?p -player ?s -stone ?ppos -location ?from -location ?to -location ?dir -direction): The player pushes the stone ?s to the location ?to. The initial position of the player was ?ppos and its final position is the initial position of the stone: ?from.

C.4 HTN Planning Domains

C.4.1 Blocksworld

Description HTN version of the classical STRIPS planning domain Blocksworld, where stackable blocks need to be re-assembled on a table. Using its hand, an autonomous agent can stack a block onto a block, unstack a block from a block, put down a block, or pick up a block.

Predicates

• (on ?x -block ?y -block): Thee block ?x is on the block ?y.

• (ontable ?x -block): The block ?x is on the table.

• (clear ?x -block): The block ?x is clear, i.e. there no block on ?x and it is not held .

• (holding ?x -block): The block ?x is held by the autonomous agent.

• (handempty): The autonomous agent does not hold any block.

Compound Tasks

• (do-put-on ?x -block ?y -block): Put ?x on ?y.

• (do-on-table): Put ?x on the table.

• (do-move ?x -block ?y -block): Move ?x on ?y.

• (do-clear ?x -block): Clear block ?x.

Primitive Tasks

• (pick-up ?x -block): The autonomous agent picks a block placed on the table. • (put-down ?x -block): The autonomous agent puts down a block on the table. ( not ( c l e a r ? x ) )

( not ( handempty ) )

( not ( on ? x ?y ) ) ) ) )

Methods

• (m0-do-put-on ?x -block ?y -block): ?x is already on ?y.

( :method m0-do-put-on • (m1-do-put-on ?x -block ?y -block): To put ?x on ?y, the agent clears ?x and ?y and moves ?x on ?y.

( :method m1-do-put-on (do-move ? x ?y ) ) )

• (m2-do-on-table ?x -block ?y -block): If ?x in on ?y, the agent unstacks ?x from ?y and puts down it on the table.

( :method m2-do-on-table

( not ( o n t a b l e ? x ) ) ) :ordered-subtasks ( and ( unstack ? x ?y ) ( put-down ? x ) ) )

• (m3-do-on-table ?x -block): ?x is already on the table .   ( :method m3-do-on-table : p • (m4-do-move ?x -block ?y -block): If ?x is on the table, the agents picks up ?x and stacks it on ?y;

( :method m4-do-move • (m5-do-move ?x -block ?y -block): If ?x is on ?z, the agent unstacks ?x from ?z and stacks it on ?y.

( :method m5-do-move • (m7-do-clear ?x -block ?y -block): If ?y is on ?x, the agents clear ?y and puts it on the table.

C.4.2 Gripper

Description In this domain, there are a robot called robby with grippers and a set of rooms containing balls. The goal is to transport balls from a given room to another.

Predicates

• (at-robby ?r -room): The robot robby is in the room ?r.

• (at ?b -ball ?r -room): The ball ?b is in the room ?r.

• (free ?g -gripper): The Robby's gripper ?g is free.

• (carry ?b -ball ?g -gripper): Robby is carrying the ball ?b with its gripper ?g.

Compound Tasks

• (move-two-balls ?b1 -ball ?b2 -ball ?r -room): Move balls ?b1 ?b2 in ?r.

• (move-one-ball ?b -ball ?r -room): Move ball ?b in ?r.

• (goto ?r -room): Go to room ?r.

Primitive Tasks

• (move ?from -room ?to -room): Robby goes to the room ?to. ( not ( at-robby ? from ) ) ) )

• (pick ?b -ball ?r -room ?g -gripper): Robby picks the ball with its gripper. ( not ( a t ? b ? r ) ) 10 ( not ( f r e e ?g ) ) ) )

• (drop ?b -ball ?r -room ?g -gripper): Robby drops the held ball. ( not ( c a r r y ? b ?g ) ) ) ) )

Methods

• (move-two-balls-0 ?b1 -ball ?b2 -ball ?r -room ?rb -room ?g1 -gripper ?g2gripper)): To move ?b1 and ?b2 in the room ?rb. Robby goes to ?ra, picks the balls, goes to ?rb and drops the balls.

( :method move-two-balls-0

: p a r a m e t e r s ( ? b1 ? b2b a l l ? r ? rbroom ? g1 ? g2g r i p p e r ) : t a s k ( move-two-balls ? b1 ? b2 ? r )

: p r e c o n d i t i o n ( and

( a t ? b1 ? rb ) ( a t ? b2 ? rb ) ) :ordered-subtasks ( and ( goto ? rb ) ( p i c k ? b1 ? rb ? g1 ) ( p i c k ? b2 ? rb ? g2 ) ( move ? rb ? r ) ( drop ? b1 ? r ? g1 ) ( drop ? b2 ? r ? g2 ) ) )

• (move-two-balls-0 ?b1 -ball ?b2 -ball ?r -room ?rb -room ?g1 -gripper ?g2 -gripper)): To move ?b in the room ?rb. Rooby goes to ?ra, picks the ball, goes to ?rb and drops the ball. • (goto-0 ?r -room): Robby is already at ?r. • (goto-1 ?from -room ?to -room): Robby goes to the room ?to.

C.4.3 Zenotravel

A transportation domain involves transporting people around in planes.

Description

Predicates

• (at ?t -thing ?c -city): The thing ?t is at the city ?c.

• (in ?p -person ?a -aircraft): The person ?p is in the aircraft ?a.

• (fuel-level ?a -aircraft ?l -flevel): The fuel level ?l for the aircraft ?a.

• (next ?l1 -flevel ?l2 -flevel): If the flevel is ?l1 then the next level will be ?l2.

• (max ?l -flevel): The max level of fuel is ?l.

• (min ?l -flevel): The min level of fuel is ?l.

• (not-min ?l -flevel): The min level of fuel is not ?l.

Compound Tasks

• (transport-person ?p -person ?c -city): Transport ?p at ?c.

• (transport-aircraft ?a -aircraft ?c -city): Transport ?a at ?c.

Primitive Tasks

• (board ?p -person ?a -aircraft ?c -city): Passenger ?p boards in ?a at the city ?c. • (debark ?p -person ?a -aircraft ?c -city): Passenger ?p debarks from ?a at the city ?c. ( not ( i n ?p ? a ) ) ) )

• (fly ?a -aircraft ?c1 -city ?c2 -city ?l1 -flevel ?l2 -flevel): ?a flies from ?c1 to ?c2. Its fuel reserves are reduced from ?l1 to ?l2. ( not ( a ta i r c r a f t ? a ? c1 ) ) 11 ( a ta i r c r a f t ? a ? c2 )

12

( not ( f u e ll e v e l ? a ? l 1 ) )

13

( f u e ll e v e l ? a ? l 2 ) ) )

• (refuel ?a -aircraft ?l -flevel ?l1 -flevel): ?a fills up with fuel in the city ?c. Its fuel reserves are increased from ?l to ?l1. ( not ( f u e ll e v e l ? a ? l ) ) ) ) )

Methods

• (transport-person-0 ?a -aircraft ?c1 -city ?c2 -city ?p -person): ?a transports ?p from ?c1 to ?c2. First of all, ?a goes to ?c1, then ?p boards in ?a, then ?p goes to ?c2 and finally ?p debarks from ?a.

( :method transport-person-0

: p a r a m e t e r s ( ? aa i r c r a f t ? c1 ? c2c i t y ( not ( a t ?p ? c2 ) ) ) :ordered-subtasks ( and ( t r a n s p o r ta i r c r a f t ? a ? c1 ) ( board ?p ? a ? c1 ) ( t r a n s p o r ta i r c r a f t ? a ? c2 ) ( debark ?p ? a ? c2 ) ) )

• (transport-person-1 ?c -city ?p -person): ?p is already in its destination.

( :method transport-person-1 : p a r a m e t e r s ( ? cc i t y ?pperson ) : t a s k ( transportperson ?p ? c ) : p r e c o n d i t i o n ( a t ?p ? c ) :ordered-subtasks )

• (transport-aircraft-0 ?a -aircraft ?c -city): ?a is already in its destination.

( :method t r a n s p o r ta i r c r a f t -0 : p a r a m e t e r s ( ? aa i r c r a f t ? cc i t y ) : t a s k ( t r a n s p o r ta i r c r a f t ? a ? c ) : p r e c o n d i t i o n ( a t ? a ? c ) :ordered-subtasks )

• (transport-aircraft-1 ?a -aircraft ?c -city ?other -city ?l1 -flevel ?l2 -flevel): ?a flies from ?other to ?c.

( :method t r a n s p o r ta i r c r a f t -1

: p a r a m e t e r s ( ? aa i r c r a f t ? c ? o t h e rc i t y ? l 1 ? l 2f l e v e l ) : t a s k ( t r a n s p o r ta i r c r a f t ? a ? c )

: p r e c o n d i t i o n ( and

( not ( a t ? a ? c ) )

( a t ? a ? o t h e r )

( f u e ll e v e l ? a ? l 1 )

( next ? l 2 ? l 1 )

( not-min ? l 1 ) ) :ordered-subtasks ( f l y ? a ? o t h e r ? c ? l 1 ? l 2 ) )

• (transport-aircraft-2 ?a -aircraft ?c -city ?other -city ?l1 -flevel ?lmax -flevel):

?a flies from ?other to ?c. Before its departure, ?a fills his fuel tank.

( :method t r a n s p o r ta i r c r a f t ( max ? lmax ) ) :ordered-subtasks ( and ( r e f u e l ? a ? l 1 ? lmax ) ( t r a n s p o r ta i r c r a f t ? a ? c ) ) )

C.4.4 Childsnack

Description This domain plan how to make and serve sandwiches for a group of children in which some are allergic to gluten.

Predicates

• (at-kitchen-bread ?b -bread-portion): The bread portion ?b is at the kitchen.

• (at-kitchen-content ?c -content-portion): The content portion ?c is at the kitchen.

• (at-kitchen-sandwich ?s -sandwich): The sandwich ?s is at the kitchen.

• (no-gluten-content ?c -content-portion): The content portion ?c is gluten free.

• (no-gluten-bread ?b -bread-portion): The bread portion ?b is gluten free.

• (ontray ?s -sandwich ?t -tray): The sandwich ?s is on ?t.

• (no-gluten-sandwich ?s -sandwich): The sandwich ?s is gluten free.

• (allergic-gluten ?c -child): The child is allergic to gluten.

• (not-allergic-gluten ?c -child): The child is not allergic to gluten.

• (served ?c -child): The child has been served.

• (waiting ?c -child ?p -place): The child wait for its sandwich at ?p.

• (at ?t -tray ?p -place): ?t is at ?p.

• (not-exist ?s -sandwich): The sandwich has not yet been prepared.

Compound Tasks

• (serve ?c -child): Make a sandwich for the child ?c.

Primitive Tasks

• (make-sandwich-no-gluten ?s -sandwich ?b -bread-portion ?c content-portion): Make a sandwich with gluten free bread and content. ( not ( n o te x i s t ? s ) ) ) )

• (make-sandwich ?s -sandwich ?b -bread-portion ?c content-portion): Make a sandwich with bread and content containing gluten. ( not ( atkitchencontent ? c ) )

( at-kitchen-sandwich ? s )

( not ( n o te x i s t ? s ) ) ) )

• (put-on-tray ?s -sandwich ?t -tray ?kitchen -place): Put a sandwich on the tray of the kitchen. ( not ( at-kitchen-sandwich ? s ) ) ( ontray ? s ? t ) ) )

• (serve-sandwich-no-gluten ?s -sandwich ?c -child ?t -tray ?p -place): Serve a gluten free sandwich to a child.

Methods

• (serve-0 ?c -child ?s -sandwich ?p2 -place ?t -tray ?b -bread-portion ?contcontent-portion ?kitchen -place): Prepare and serve a gluten free sandwich to an allergic child. ( no-gluten-content ? c ont ) ) :ordered-subtasks ( and ( make-sandwich-no-gluten ? s ? b ? c on t ) ( put-on-tray ? s ? t ? k i t c h e n ) ( move-tray ? t ? k i t c h e n ? p2 ) ( serve-sandwich-no-gluten ? s ? c ? t ? p2 ) ( move-tray ? t ? p2 ? k i t c h e n ) ) )

• (serve-1 ?c -child ?s -sandwich ?p2 -place ?t -tray ?b -bread-portion ?cont -content-portion ?kitchen -place): Prepare and serve a sandwich to a nonallergic child. ( not ( no-gluten-content ? c ont ) ) ) :ordered-subtasks ( and ( make-sandwich ? s ? b ? co nt ) ( put-on-tray ? s ? t ? k i t c h e n ) ( move-tray ? t ? k i t c h e n ? p2 ) ( serve-sandwich ? s ? c ? t ? p2 ) ( move-tray ? t ? p2 ? k i t c h e n ) ) )

C.4.5 Transport

Description A transport domain where each vehicle can transport some packages depending on its capacity and moving has a cost depending on the length of the road

Predicates

• (road ?l1 -location ?l2 -location): There are a road between ?l1 and ?l2.

• (at ?x -locatable ?l -location): ?x is at ?l.

• (in ?p -package ?v -vehicle): ?p is in ?v.

• (capacity ?v -vehicle ?n -capacity-number): The capacity of ?v is ?n.

• (capacity-predecessor ?n0 -capacity-number ?n1 -capacity-number): ?n1 = ?n0 -1.

Compound Tasks

• (deliver ?p -package ?l -location): Deliver ?p at ?l.

• (get-to ?v -vehicle ?l -location): Go to location ?l with ?v.

• (load ?v -vehicle ?l -location ?p -package): Load ?p in ?v at ?l.

• (unload ?v -vehicle ?l -location ?p -package): Unload ?p from ?v at ?l.

Primitive Tasks

• (drive ?v -vehicle ?l1 -location ?l2 -location): ?v goes to ?l2 from ?l1. A road connecting ?l1 to ?l2 is required. ( not ( a t ?v ? l 1 ) ) 9 ( a t ?v ? l 2 ) ) )

• (pick-up ?v -vehicle ?l -location ?p -package ?s1 -capacity-number ?s2capacity-number): Pick ?p in ?v. ( not ( c a p a c i t y ?v ? s2 ) ) ) )

• (drop ?v -vehicle ?l -location ?p -package ?s1 -capacity-number ?s2 -capacitynumber): Drop ?p from ?v. ( unload ?v ? l 2 ?p ) ) )

• (load-0 ?l -location ?p -package ?s1 -capacity-number ?s2 -capacity-number ?v -vehicle): Load ?p in ?v. • (get-to-0 ?l -location ?v -vehicle): ?v is already at ?l. • (get-to-1 ?l1 -location ?l2 -location ?v -vehicle): ?v goes to a location ?l1 connected to ?l2, and moves from ?l1 to ?l2.

D.1 Introduction

Le domaine de l'intelligence artificielle (IA) vise à développer des agents autonomes capables de percevoir, d'apprendre et d'agir sans aucune intervention humaine pour accomplir des tâches complexes. Comme l'indique [START_REF] Russell | Artificial intelligence: a modern approach[END_REF], un agent autonome est un système intelligent qui peut décider de ce qu'il faut faire pour réaliser une tâche. Un agent autonome perçoit son environnement, planifie les meilleures actions possibles et les exécute. Pour planifier les meilleures actions, l'agent autonome doit donc être capable de prendre des décisions. Il existe plusieurs approches permettant aux agents autonomes de prendre des décisions. L'une d'entre elles est la planification automatique [START_REF] Fikes | STRIPS: A new approach to the application of theorem proving to problem solving[END_REF][START_REF] Ghallab | Automated Planning: Theory and Practice[END_REF].

D.1.1 Planification automatique

L'objectif de la planification est la résolution de problèmes de planification. La description d'un problème de planification se fait de manière déclarative. La déclaration d'un problème de planification est composée de : (1) l'état initial de l'environnement, (2) un but et (3) l'ensemble des actions qui peuvent