C Restrictions

A Modified Planning Algorithm for SAS + -PUS Problems B Domain Operator Graphs with (PU) Operators

Attack(6)

AttackGrenade

GrenadeFromCover(2)

) Ready(7)

AttackFromNode

Crouch(5) FromView(4.5)

)

) has the following properties:

1. G O (v i ) is weakly connected.

1

Introduction

This thesis is about Artificial Intelligence (AI) Action Planning. In particular, my researches focus on action planning in real-time and I use video games to test, experiment and analyze my findings. AI is an area of knowledge and research aiming to design intelligent computational agents, i.e. agents that act intelligently in their environment and according to their goals. In my opinion, the best definition of AI, or Computational Intelligence, is the one of D. Poole, A. Mackworth and R. Goebel [1, p.1]:

Computational intelligence is the study of the design of intelligent agents. An agent is something that acts in an environment -it does something. Agents include worms, dogs, thermostats, airplanes, humans, organizations, and society. An intelligent agent is a system that acts intelligently: What it does is appropriate for its circumstances and its goal, it is flexible to changing environments and changing goals, it learns from experience, and it makes appropriate choices given perceptual limitations and finite computation. The central scientific goal of computational intelligence is to understand the principles that make intelligent behavior possible, in natural or artificial systems. The main hypothesis is that reasoning is computation. The central engineering goal is to specify methods for the design of useful, intelligent artifacts. This introduction is organized as follows: I first give a definition of AI planning in realtime, I then give my motivation followed by my approach to answer the main question: how to reduce time complexity to scale up real-time planning? I end the introduction with the outline of my thesis.

AI Planning in Real-Time

An AI planning problem, or more precisely an AI action planning problem (to distinguish with pathfinding and other sub-fields of planning), is an AI problem where an intelligent agent can plan in its environment to reach its goals. To do so, the environment and relevant parameters of the agent are symbolized to form states that the agent uses to evaluate a real situation at a given time in an abstract way. Most of the time, an agent states its current situation or its goals. The agent is then equipped with actions or operators that the planner uses to create action plans in order to look ahead in time and perform actions in the world. Furthermore, actions and action plans both act as a transition function between two states. They can have a cost in order to characterize the quality of a transition, i.e. the relevance of an action or an action plan for a given situation.

Planning in real-time concerns planners that plan in a very short amount of time, namely in a millisecond or two at most. A good example of intelligent agents using such planners are the Non-Playable Characters (NPCs) populating some video games [2][3][4][5]. For those games, real-time means 10% of the time between two frames1 , i.e. 3.33ms for a video game with a framerate of 30 Frame per Second (FPS), 1.67ms for a 60 FPS games, 0.83ms for a 120 FPS game, and so on. As I am interested in NPCs controlled by planners, I will present the Goal-Oriented Action Planning system as it is the technology used in today's video games to plan in real-time (cf. Chapter 2). According to a study I conducted on GOAP, I found patterns that can be used to restrict planning problems. These patterns lead me to study the Simplified Action Structure formalism [6] which I present in Chapter 3. I then use this formalism in Chapter 4 and Chapter 5 as a basis for my work.

Motivation

I strongly believe planning is a field of AI that is the future for NPCs behaviour. Video games are more and more realistic and beautiful due to the graphics. Maps move from corridors (Call Of Duty, The Witcher 1 & 2, Pacman...) to huge open-worlds (The Elder Scrolls: Skyrim, Gran Theft Auto, Assassin's Creed, Red Dead Redemption,...), and even infinite open-worlds thanks to procedural generation (Minecraft, No Man's Sky...). Some stories are so immersive that you feel as if you are an actor in a movie (The Witcher 3, Detroit: Become Human, Fable: The Lost Chapter,...). Virtual Reality headsets are close to be found in every household, especially with the announcement of the Metaverse. That is to say, there are many new technologies to create more and more immersive video games. This is therefore important to populate these high quality environments with believable NPCs, i.e. NPCs that do not only patrol or interact purposelessly with the player.

Believable NPCs are NPCs that live in the environment, that seem to have a real life no matter what the player does. If the NPCs have to interact with the player, their actions have to be meaningful according to their goals. That is where planning is interesting, depending on the goal of one NPC, the planner will try to create an action plan that links the current state of the NPC to the goal state. An action plan is more meaningful than just one action in a sense that the first action of the plan is useful for one or more actions that will come next. As an example, an NPC that carries a box may do that to strengthen a position, and the same NPC will then use his strengthened position to take cover so as to prepare an ambush and so on. Goal-Oriented Action Planning (GOAP) [7] is the most used planner in video games. F.E.A.R. [2], Shadow of Mordors [3], Assassin's Creed [5], Rise of the Tomb Raider [4], all these games have used GOAP to make their NPCs more believable. NPCs of F.E.A.R are still perceived as some of the best NPCs so far in the video game industries [8]. Pedestrians, faunas, enemies of recent Assassin's Creed (Odyssey, Valhalla...) are driven by GOAP: it makes these games more dynamic and immersive for the players [9].

Real-time means the planner has to provide as many action plans as possible within the time budget, e.g. between two frames in a video game. The more agents to control, the less time to generate one plan. Idem for the number of actions inside a plan, the more actions, the more time to generate one plan. The worst-time complexity is therefore a significant criteria to evaluate the planning algorithm. Unfortunately, the current planning system used in GOAP is search-based and has an exponential time complexity [7]. This leads to development restrictions to respect the time budget while trying to meet the growing expectations of the players. My main question throughout this thesis is therefore: how to reduce time complexity to scale up real-time planning?

Approach

My approach to answer my main question was to first study a GOAP system inside a commercial video game. F.E.A.R., which is still a great video game reference in term of believable NPCs, is also one of the few commercial video games using a GOAP system and whose source code is freely available online [10]. Based on the results of my AI planning analytics of F.E.A.R., I then analyzed several dozen GOAP GitHub projects to see whether the planning system is always implemented the same way. I finally reached some Lead AI and teams that implement GOAP in their video games to ask them some questions about their development habits. Eventually, I drew conclusions that I then turned into assumptions.

These assumptions then led me to study the Simplified Action Structure (SAS) formalism. Indeed, two of my assumptions are in fact existing input restrictions of this formalism. C. Bäckström, the founder of SAS, restricted SAS planning problems to find tractable subclasses of problems. He also developed a polynomial time algorithm that solves instances of his tractable problems. I re-used C. Bäckström's approach to create new classes of problems adapted to NPCs in video games and I then developed a linear-time algorithm to solve these new problems. Eventually, I carried out several experiments using several different benchmarks to evaluate the effectiveness of my algorithm and to check that it indeed respects the real-time constraint.

Outline of the Thesis: my Plan.

Chapter 2: Goal-Oriented Action Planning This Chapter presents the GOAP system, my planning analytics on F.E.A.R. and my findings. The conclusions allowed me to define Proposition 2.4.1 which presents my assumptions for the rest of the manuscript.

Chapter 3: Simplified Action Structure I studied the work of C. Bäckström [6] to answer my main question and to create a planner that can plan in real-time for more than a few dozen of NPCs and with longer plans. I re-use C. Bäckström's approach to reduce the time complexity. He restricted the input of planning problems to create tractable ones that can be solved by a polynomial time algorithm. Before going any further in his direction, I recall the theory behind SAS and adapt it to my notations and my needs. I present the classes of problems SAS-PU and SAS + -PUS. The latter is a class of tractable problems whose instances can be solved by an existing polynomial time algorithm. I implemented it in C++ and tested it on realistic benchmarks.

Chapter 4: Topological Planning This chapter is the core of this thesis where I present new classes of problems and my linear-time algorithm, named TopoPlan and denoted P, that solves instances of these new problems. In this chapter, I first present prerequisites that take advantage of the post-uniqueness and unariness restrictions. In particular, I present how to identify operators and actions in constant time while planning and how the identifiers can be used to create new sets of predecessors. Then, I introduce operator graphs and action graphs which are graphical representations of the input and the output of a SAS-PU planning problem. I then use these graphs to prove the Υ 1 -Truth Criterion, a truth criterion that take into account my assumptions, and to introduce a new structural restriction, denoted C k , which allows me to create new classes of problems respecting my assumptions. I finally present my linear-time algorithm that takes advantage of the prerequisites and which I prove correct and complete with the previously the Υ 1 -Truth Criterion.

Chapter 5: Benchmarks and Results

This chapter is divided in two parts followed by a discussion. The first section presents scalable benchmarks which are benchmarks designed to grow with respect to the number of state variables or with respect to the maximum number of values in a state variable domain (cf. M and D v i in Definition 3.1.1). Among these benchmarks are the BlockWorld and the Tunnel examples [6]. I use these benchmarks to strengthen empirically my theoretical results. The purpose of the second section is to show that realistic benchmarks can be represented with my new classes of tractable problems. Among those realistic benchmarks are RDR-2, ACO and HZD which are respectively based on the commercial video games Red Dead Redemption [11], Assassin's Creed: Origins [12] and Horizon Zero Dawn [13]. I then use these benchmarks to compare my C++ implementation of both P and P. The results are astonishing: P is about > 30 thousands times faster than P to solve instances of these reaslistic problems. I end the chapter with a discussion that gives explanations on why P is much faster than P as well as some other interesting differences between the two. Chapter 6: Conclusion I eventually conclude my thesis by summarizing each chapter and, then, by answering my main question: how to reduce time complexity to scale up real-time planning? I then present some possible future works based on my thesis results.

2

Goal-Oriented Action Planning

Planning in real-time is one of the alternatives [14] for modeling and editing the behavior of Non-Player Characters (NPC) in commercial video-games. Goal-Oriented Action Planning (GOAP) was first introduced in F.E.A.R. [2], a First-Person Shooter (FPS) released in 2005. When the goal of an NPC is updated according to the current game state, GOAP searches for a plan in a backward manner to achieve this new goal: GOAP starts with searching an action which can achieve this new goal in the set of actions of the NPC; the preconditions of such an action become new goals to achieve, and so on until the current game state can satisfy the preconditions of an action. During the search, context preconditions are used to query of the current game state and compute various numerical values (e.g. distances to dodge).

In F.E.A.R., actions are implemented as C++ classes, and search reuses A* [15] from the path planning module of the game engine; the heuristic function used for GOAP evaluates each search node summing the costs of actions, and then computes the number of goals yet to be satisfied. Action costs in F.E.A.R. are not dynamic (on the contrary of the implementation of GOAP in the Tomb Raider series released by Crystal Dynamics since 2013 [4], for instance): once designed and implemented in the game, action costs are constant values that will never be modified during the search.

In this chapter, I address the use of action costs in GOAP, and more generally the representation of actions in practice in commercial games using GOAP. Indeed, this system handles unexpected game situations better than other behavior engineering alternatives; but game designers feel a lack of control [14] over an AI architecture able to produce unexpected NPC behaviors [7]. Consequently, action costs appear as a mean of tweaking what happens on-screen, thus entailing the recurring question of a new to GOAP development team: how to design the action costs?

F.E.A.R.'s AI has been widely recognized as the best in class when released [16], and still is recognized as such [8]. As controversial as such reviews can be, F.E.A.R.'s GOAP nevertheless has had a great impact among game AI developers and definitely sparked the use of planning in commercial video-games in general and in FPSes in particular.

Thanks to the SDK released by Monolith in 2006 [10], the game AI development dif-Figure 2.1: C++ action hierarchy for attack actions of the action set of the basic soldier in F.E.A.R. where each node represent a class; the root is the Abstract node colored in yellow. The cost of each action appears in brackets after the name of the action. Actions whose names appear in italics are abstract C++ classes that cannot be instantiated. All actions except Ready follow Jeff's proposal of assigning lower costs to the leaves of the class hierarchy.

ficulty is no longer a coding problem: the C++ code of F.E.A.R.'s GOAP can be studied and its main parts can be duplicated. Although today many games use GOAP for NPC behavior engineering, F.E.A.R. remains to this day the only game whose SDK allows access to the C++ code of the planner. There are also more than 300 GOAP repositories on Github, implementing planning functionality for potential commercial video-games in python, ruby, and javascript. C++ and C# share 54 implementations, not all of which will end up in a commercial game, of course; but if that's the case, then these planners will face what is now the crucial step of the use of GOAP in a video-game: the design of the action costs for the game for which they have to generate plans. Indeed, using GOAP for a new game means new actions, and consequently means new action costs must be designed.

The original author of GOAP, Jeff Orkin, proposes to "apply a cost to actions to force A* to consider more specific actions before more general ones." [7, page 108]: that is, the higher the action in the C++ inheritance hierarchy, the larger the action cost; and that is indeed what I observe when studying F.E.A.R.'s SDK. Figure 2.1 shows the distribution of the action costs in F.E.A.R.. GOAP uses A* to search for plans: when a solution is found, its cost merely is the sum of the cost of its actions, as all goals of a solution-plan are satisfied; therefore, this cost is the minimal cost that can be found. At each step during search, when faced with a choice among several actions, GOAP will choose the one with the lowest action cost to achieve the minimal cost for a solution. We can observe that this is not what happens while playing F.E.A.R., however; not only any kind of expensive action can appear more often than a lower cost action, but this is also the case if I restrict my study to actions achieving the same goal. Action usage (%)

Action cost

In F.E.A.R., low-cost actions are not necessarily used more than higher-cost actions x-axis represents the action usage and y-axis is the action cost; the 72 (red) dots of this graphic represent the 72 actions logged in the F.E.A.R. sessions. We can observe that there is no clear influence of the action cost over the action usage: some, more expensive, actions are used much more than other, less expensive, actions.

sessions from 10 players where 337600 plans were generated and recorded so that action occurrences could be computed; each red dot represents one of the 72 actions logged in the in-game data, whose abscissa indicates its usage and the ordinate indicates its cost. We can observe that the right-most red dot, which represents the most used action in the game (i.e. this action occurs about 22.5% of the time), has a cost of 3 which is much higher than the cost of much less used actions that appear in the bottom left corner of the figure.

Indeed, we can observe in Figure 2.4 the following: for any pair of actions (a 1 , a 2 ) such that action a 2 occurs more than action a 1 (that is, action a 2 appears more on the right in Figure 2.2 than action a 1 ) then the average value of cost(a 2 ) cost(a 1 ) is 1.67. Consequently, (1) despite the use of A* and the search for minimal cost plans, expensive actions occur more often than cheap actions, and (2) more abstract actions occur more often than more specific actions, despite Jeff Orkin's proposal.

If I restrict my study to attack actions achieving the same KillEnemy goal, for instance, Figure 2.3 shows the same result: some expensive attack actions are used more than attack actions with a lower cost, which definitely brings to the forefront the question of how to design the costs of actions.

If game designers cannot rely on action costs to impact action usage, then why use action costs at all? Are action costs useful for real-time purposes or for anything else in GOAP? If the use of action costs in GOAP is unclear, can we safely remove action costs? In the next sections I conduct a close study of the action costs in GOAP in order to provide answers to these questions ; I begin with explaining GOAP. Attack action usage (%)

Attack action cost

In F.E.A.R., low-cost attack actions are not necessarily used more than higher-cost attack actions x-axis represents the action usage and y-axis is the action cost; We can observe that there is no clear influence of the action cost over the action usage: some, more expensive, attack actions are used much more than other, less expensive, attack actions.

The Theory of GOAP

GOAP is a decision-making system coupled with a planning one. The world representation of an agent is based on STRIPS: it is an action-centric representation but with multi-value state variables, and each action has a cost. Then, the agent has to be equipped with sensors, a set of goals and a set of actions which will allow the NPC to sense, plan and act in its environment. Each time a perturbation occurred, the agent will sense it and then perform a goal relevance calculation thanks to its decision-making system to decide what to do. It will then decide how to achieve its goal thanks to its planning system which will search for an action plan by using the A* search algorithm. Finally, all of this must be achieved in a short period of time in order to respect the constraint of agent control in real-time. In this section, I illustrate my explanations with examples from the video game F.E.A.R.. I also explain how GOAP is implemented inside this game in subsection 2.1.3.

The Action Representation

As GOAP is STRIPS-based, the actions are represented with the classical planning model, that is with pre-and post-conditions that are conditions on state variables. This actioncentric representation coupled with a search algorithm will allow the planner to build sequences of actions in order to build solutions plans and, therefore, to link two different states. The following will present how these parameters are used. In F.E.A.R., actions used more often than others also cost 1.67 times more on average. Figure 2.4: For all pairs of actions (a 1 , a 2 ) such that a 1 is used less than a 2 , the action cost ratio is cost(a 2 ) cost(a 1 ) . The median value of the action cost ratio, represented by the while line, is 1: 50% of the action cost ratios are ≤ 1 while 50% of the action cost ratios are ≥ 1; the mean value of the action cost ratio, represented by the black line is 1.67: any action a 2 used more than a 1 is on average 1.67 more expensive than a 1 . If we restrict the study to attack actions from Figure 2.1 then the average value is 2.1.

The postconditions

They are used to satisfy goals or unsatisfied preconditions of other actions (these preconditions can be seen as sub-goals). Then, as A* goes backward to search for actions (see 2.1.2 and Algorithm 1 for more details), the first pruning is made by selecting actions with the proper postconditions. Only these actions can become candidates in the search space of A*. For example in F.E.A.R., for the soldier, if KillEnemy is called, the goal is to set the state symbol T argetIsDead to the value T rue. Thus, the first step for the planner is to only keep actions having the postcondition T argetIsDead = T rue. For the soldiers, which are the most basic NPCs in F.E.A.R., 10 actions possess this postcondition. Given the fact that A* goes backward to search, this means the brute branching factor at this node is 10. So, the more actions with the same and only postcondition, the higher the brute branching factor is to satisfy a goal or an unsatisfied precondition. The average branching factor is never equal to the brute one due to context preconditions, however. (See 2.1.4 for more details). It is also worth noticing that actions are unary in F.E.A.R., i.e. each action only have one post-condition. This can be seen in the source code provided by the SDK.

The preconditions

When an action is chosen during the plan building process, its preconditions are then consulted to figure out the next step. All its unsatisfied preconditions become goals to achieve. That is, this action can not be activated by the NPC at its current state, therefore other actions have to be considered. The process continue until all goals is achieved. One should notice that the more actions whose preconditions are unsatisfied, the greater the depth of search, i.e. the greater the number of states to be crossed during the search.

The Search Algorithm

Originally, i.e. in the video game F.E.A.R., the search algorithm used to search for valid plans is A* [7]. It was already implemented for the pathfinding: its reuse avoided the implementation of a new search algorithm, and reduced both development time and bugs. It is the same for other commercial games such as Middle-earth: Shadow of Mordor [3] or Rise of the Tomb Raider [4]: they use A* in their GOAP planning system. On GitHub, I have analyzed the source code of all the C++ and C# projects, which represent 54 projects. Only half of them uses A*. A third of them do not use any search algorithm: they build the entire graph before selecting the cheapest path. The remaining ones use a best-first search algorithm. Due to the preponderance of A* in the various GOAP systems, and because it is the original one chosen by Jeff Orkin, I decided to only considered A* in my study.

A* is used to search for valid plans. Game developers had to configure the actions and the planner properly so as to use together [7]. A* works with heuristics and cost metrics. At the frontier of the search (See SF on Algorithm 1), the node with the lowest score will be selected. The score, returned by the fitness function f , is the sum of the heuristic function h and the cost function g: h estimates the remaining distance from a node at the search frontier to the goal node whereas g evaluates the current cost to reach this node from the start. To use A* in GOAP, a node is characterized by its goals (a subset of a goal state, noted G in Algorithm 1) and the action plan built to reach itself (noted P ); a move from a node to another is fulfilled thanks to an action. The selection criterion for the next node to explore is still made by the fitness function f : the one with the lowest score is selected (noted min(SF )). To evaluate f , the cost and the heuristic functions had to be adapted: the first one evaluates the sum of the action costs of a plan (g(P )), while the second one counts the number of goals (h(P )). The output of the algorithm is an action plan, and it will be the cheapest one if A* is admissible [17].

In F.E.A.R., most of the actions have only one postcondition, therefore the heuristic does not only evaluate the number of goals, it also evaluates the number of actions required to achieve the goals of a node: if a node possesses two goals, then at least two actions is required. Then, all the action costs1 are equal to or greater than 1.0, therefore the heuristic is admissible and it is optimal in F.E.A.R.: the planner always returns the best solution [17], i.e. the cheapest one. One should notice that the cheapest plan does not mean the shortest one. Indeed, let imagine two plans P 1 and P 2 that solve a goal from the NPC's current state, if P 1 is composed of three actions with a cost of 1 (so g(P 1 ) = 3) while the second one is composed of one action with a cost of 10 (so g(P 2 ) = 10), then the cheapest plan returned by A* is P 1 and it is not the shortest one (3 actions against 1). Thus, in F.E.A.R., as the heuristic function h also estimates the number of actions to use, Algorithm 1 A* for GOAP {The search for a plan goes backward from the goals to the NPC's current state.} Inputs: G = ∅, a list of goals; A = ∅, the NPC's action set; S C , the NPC's Current state; Output: P , an action Plan. P ← ∅ g(P ) ← 0 {the cost function of P } h(P ) ← Length(G) {the heuristic function of P } f (P ) ← g(P ) + h(P ) {the fitness function of P } SF ← {(G, P )} {min(SF ) is the couple (G, P ) ∈ SF (Search Frontier) such that ∀(G i , P i ) ∈ SF, f (P ) ≤ f (P i ).} while SF = ∅ do (G i , P i ) ← min(SF ) if G i ∈ S C then Return P i {Exit with success} else for each action a ∈ A whose at least one postcondition belongs to G i do P j ← {a} + P i G j ← G i \ P ost(a) for each precondition pre of a do if pre / ∈ S C and pre / ∈ G j then G j ← G j + {pre} end if end for g(P j ) ← g(P i ) + Cost(a) h(P j ) ← Length(G j ) f (P j ) ← g(P j ) + h(P j ) SF ← SF + {(G j , P j )} end for end if SF ← SF \ {(G i , P i )} end while Return ∅ {Exit with failure} it fosters short plans. Indeed, the more action required, the greater the heuristics. And if h increases, f also increases (f = g + h, g being independent of h), which results in not favoring nodes that lead to long action plans (according to the selection criterion of A*). In my previous example, if we focus on the first step of the search, for each plan, P 1 1 and P 1 2 (where P 1 2 = P 2 ), we have 2 ≥ h(P 1 1 ) ≥ 1 whereas h(P 2 2 ) = h(P 2 ) = 0. Therefore, the heuristic fosters P 2 , the shortest plan and not P 1 the cheapest one.

The Case of FEAR

In the video game F.E.A.R., sensors (e.g. line of sight, detecting light, hearing footsteps, etc.) allow the Non-Playable Characters (NPCs) to have an individual understanding of their surroundings. The acquired pieces of information are stored in their working memory and if it is a shareable knowledge, it is then managed in a blackboard [7]; this technique allowed developers to minimize the number of the state variables for the NPCs [7]. There are 21 different state variables in F.E.A.R. like TargetIsAimingAtMe, TargetIsDead, AtNodeType or WeaponLoaded. These two components (the individual understanding and the state variables) are both required by the decision-making system to assign goals to an NPC. Then, the planning system is requested to search for a valid plan to achieve the goals, as explained in subsections 2.1.1 and 2.1.2.

Although the previous implementation description is the basic one for GOAP [18], context preconditions have been added to actions so as to lighten the planning process [7].

The Context Preconditions

According to what I explained in subsections 2.1.1 and 2.1.2, actions are first selected with their postconditions. As several actions can have the same postconditions (in F.E.A.R., 84 actions are available for 21 state variables, which entails that some actions possess the same postcondition), with no other selection criterion than the postconditions, the branching factor of each node of the search space is equal to the number of actions having the same postconditions: it is the brute branching factor. In order to maintain "Lightweight Planning" [7] in F.E.A.R., context preconditions have been added to action. Their purpose is to minimize the search space and optimize precondition validation so as to maintain a fast enough planning system for real-time. Indeed, let suppose we have b actions every step and a depth of search of d, the complexity of the A* search algorithm is O(b d ). Therefore, as a developer, you definitely want to keep b and d as low as possible so as to avoid a combinatorial explosion and, consequently, a frame rate drop while planning in real time. The context preconditions, which are queries on the world, will impact b by pruning candidate actions. Each action will have their own ones and it will allow the planner to decide if the current action is viable or not. If not, then the action is merely withdrawn from the search, and so b decreases. For instance in F.E.A.R., if an NPC detects a disturbance, its goal will be to set the state symbol DisturbanceExists to the value F alse. Four actions possess this postcondition but only two of them, ReactToDanger and EscapeDanger, will be considered if the disturbance is dangerous while the other two, InspectDisturbance and LookAtDisturbance, will be considered for safer disturbances. This technique, the pruning with context preconditions, has also been used by other studios than Monolith Productions. Crystal Dynamics ("Rise of the Tomb Raider") or High Moon Studio ("Transformers 2") have also implemented it in their commercial games using GOAP [19]. Furthermore, among the GitHub projects observed, a quarter of them use this technique.

In the next sections, I study the impact of these context preconditions on planning and I am going to show that they are behind the no clear influence of the action cost over the action usage.

Results

A* is the search algorithm used in GOAP to search for plans, actions must therefore be designed with a cost: it will be used by the planner to distinguish actions that will compete for a same goal. Therefore, the proposal made by Jeff Orkin was: "apply a cost to actions to force A* to consider more specific actions before more general ones" [7], i.e. the lower the cost, the more likely an action is to be chosen. As explained in the introduction of this chapter, in the video game F.E.A.R., for attack actions having the same postcondition (KillEnnemy, i.e. T argetIsDead = true) some expensive attack actions are used more than attack actions with a lower cost. This contradicts Jeff Orkin's proposal. In fact, it seems that context preconditions, by pruning candidate actions, play a more important role in planning than expected.

The Context Precondition Impact on the Branching Factor

As I explained in subsection 2.1.1, many actions can have the same postcondition, which means for a same goal we can have several candidates: this is the brute branching factor. But, in subsection 2.1.4, I have highlighted that the context preconditions impact the branching factor (BF ) by withdrawing actions that are not viable depending on the context: indeed, BF decreases each time an action is removed. I made an experiment in F.E.A.R. in order to visualize the average branching factor while planning. The table 2.2.1 gives the influence of the context preconditions on the branching factor with respect to four of the most diverse goals, i.e. the number of actions for these goals are greater than the others. ("WSE" stands for state Event.) 19

According to the results, the context preconditions have been a strong pruning: they have greatly reduced the branching factor of the search. In F.E.A.R., a soldier can choose between 10 attack actions with T argetIsDead = T rue as their only postcondition. It implies a brute branching factor of 10 but, in practice, the branching factor has an average been equal to 1.52. In other words, the search algorithm for the goal T argetIsDead = T rue has on average had half a chance to only have one candidate action and half a chance to have two candidate actions. Therefore, the context preconditions can have a great impact on the branching factor: what is the place left for the action costs which such an impact?

Action Plan Construction

In order to see the impact of the action costs and the impact of the context preconditions on the planning activity while playing, I have carried out two experiments on the video game F.E.A.R..(1) I have asked some players to play the game entirely and several times with two different cost configurations: the first one is the original game with the original costs (called OC + configuration), while the second one set all the costs to 1, i.e. unitary costs (called UC configuration). ( 2) I have played the game entirely with the original costs but with the context preconditions removed from the planning activity (called OC -configuration). During all these gaming sessions, I have gathered data: 647,548 action plans were generated during (1): 337,600 plans come from the OC + sessions (they represent 58 hours 21 minutes of playing time) while 309,948 plans come from the UC sessions (58 hours 35 minutes) ; 754,205 action plans were generated during (2) and it only represents 4 hours 5 minutes of playing time. Among all these plans, there are only 87 distinct ones in the framework of OC + , 73 in UC and 56 in OC -. In other words, the GOAP planner in F.E.A.R. has only created 87 plans in OC + , 73 in UC and 56 in OC -and have repeated them many times during the game sessions. It corroborates results in [20] explaining that AI planning pattern can be found with GOAP. In addition, UC and OC + share 69 common plans (68 if we remove the empty plan: {}, i.e. if we only keep the successful plans) which represent 95.61% of the successful OC + plans and 99.93% of the successful UC plans. OC + and OC - share 48 common plans (47 if {} is removed) which represent 95.43% of the successful OC + plans and 87.08% of the successful OC -plans. Let P OC + , P U C and P OC -be respectively the finite set of distinct plans generated with OC + , the finite set of distinct plans generated with UC and the finite set of distinct plans generated with OC -; we have #P OC + = 87, #P U C = 73 and P OC -= 56. P OC + -P U C , P U C -P OC + , P OC + -P OC -and P OC --P OC + represent the respective complements between P OC + and P U C and between P OC + and P OC -. The table in subsection 2.1 gives information on these finite set of distinct plans with respect to the plan length (where the length is the number of actions). If we remove the planning failures from the results, which represent a large part for the UC and OC + configurations, the occurrence probability of P OC + -P U C is 4.503% (instead of 1.133%) whereas the occurrence probability of P U C -P OC + is 0.065% (instead of 0.016%). Consequently, I consider that P U C ⊂ P OC + , i.e. all the plans generated in UC were generated in OC + , i.e. the setting of the costs in F.E.A.R. has brought a diversity represented by the complement P OC + -P U C , which is 4.503% if we Furthermore, the majority of the successful plans have one or two actions. Indeed, planning failures set aside, they represent about 98.8% of the generated plans in OC + and UC, which corroborates the results in [21] showing that GOAP returns short plans.

Figure 2.5 shows that the probability of generating a plan as a function of the number of actions constituting it has been broadly the same for OC + and UC. Planning failure represents respectively 74.7% and 75.2% for both of these configurations, where a failure means the planner was not able to build a valid plan to achieve a given goal, so it returns an empty plan (See subsection 2.3.2 for more details about planning failure). Thus, to modify the costs did not affect the length of the plans. On the contrary, with OC -, only single-action plans (almost) were generated by the planner. Therefore, in OC + , context preconditions allowed the construction of longer plans. Furthermore, almost no planning failure can be seen with OC -, which means the planner never failed to find a plan. But, planning failure is not necessarily a bad thing because action plans returned with OC - were not often the most realistic ones: the NPCs behaved oddly while playing. This last analyze can be highlighted with Figure 2.6 which compares the occurrence probability of the common attack action plans between the three configurations. The most likely plan for OC -is the single action plan {AttackF romV ehicle} with an occurrence probability of 32.79%. This plan makes sense in F.E.A.R. when an NPC is close to a vehicle, but this is not often the case. On the other hand, the most likely attack action plan for UC is the single action plan {Attack} while, for OC + , it is {AttackF romCover}. There is also a slightly higher difference in favor of OC + for the other attack plans. To summarize, OC+ has been the configuration with the most diversity and consistency for the NPC's behavior, but the differences with UC are very small unlike the differences with OC -. It highlights that the context preconditions have played a significant role in the action plan construction.

Concerning the plan {AttackGrenadeF romCover} in OC -, its occurrence probability exists and is high because the action AttackGrenadeFromCover has as a second postcondition: T argetIsF lushedOut = T rue (It is one of the very few actions having more than one post-condition). Which means the planner were able to solve the goal T argetIsF lushedOut = T rue with this action. Otherwise, for the goal T argetIsDead = T rue the only possible attack action plan for the OC -configuration is {AttackFrom-Vehicle}. Indeed, the attack action costs range from 1 (AttackFromVehicle) to 7 (Attack-Ready) with the original configuration and, as attack action plans are most of the time single action plans, the heuristic functions during the search have always been equal to 0 (meaning the action can be activated in the NPC's current state), the only possible action plan to be returned is: {AttackF romV ehicle}, the cheapest one.

Concerning the other possible action plans, they were no significant differences between the OC + and the UC configurations (the greatest differences lie in the attack action plans).

On the other hand, the differences between the original game and the game without the context preconditions have been more notable and were clearly visible in game. NPCs have had inconsistent behaviors and were contextually more irrelevant. Finally, context preconditions have a significant impact compared to the costs. The context preconditions shape the search space while the use of action cost only brings a slight diversity.

Construction Time

In 2005, a video game had a frame rate of 30 FPS, which entails the time budget for one frame is 0.033s. As the computing resources have to be shared with many different modules, such as animation, rendering, and physics systems, the allocated time budget for the AI module is no more than 10% of the total time budget, i.e. 3.3ms for a 30 FPS video game. Because GOAP has the constraint to plan in real-time, i.e. between two frames, it seems obvious that any kind of optimization can be useful so that its implementation does not hinder the fluidity of the game [7]: could the action costs help to achieve real-time planning?

As we can see on Figures 2.7 and 2.8, which represent the planning runtime of the three configurations (Figure 2.8 differentiates the length of the plans), the modification of the costs, from OC + to UC, have barely influenced the runtime unlike the context preconditions. Therefore, adding context preconditions extend the planning runtime by a few tens of microseconds. It was good for the video game F.E.A.R., the context preconditions avoided unnecessary or unfeasible plans. In fact, returned plans with OC -were often not feasible, so NPCs had to ask the planner again and again for new plans, which led to frame rate drops and odd behaviors from NPCs in game. Lastly, the allocated time budget (3.3 ms -10% of a frame) has been far from being overused (0.040 ms on average), which means there is plenty of time left to create longer plans. In fact, the faced computational limits were related to memory usage [22], and not the time as explained by my results. Which means the GOAP system with these settings manages the real-time constraints.

Discussion

Through this section, I discuss previous works and bring further explanations thanks to my results (section 2.2) on the impact of the context preconditions on GOAP's plan construction.

The Context Precondition Impact on Planning

As explained in section 2.1.4, context preconditions have been designed to maintain lightweight planning [7]. They have also been used as a way to contextually keep or remove a candidate action without burdening the symbolic representation of STRIPS. Therefore, they have also been part of the planning process. There is no study yet that have examined, in practice, their impact on planning, however. I have highlighted that they have not just been part of the planning system: if they are overused, they can basically shape the whole planning activity by overtaking the use of the action costs. Figure 2.6 corroborates this observation: although the action costs have had a sensible impact on the attack action plans, the overall behavior of the search algorithm has not changed with the cost modification. In 2007, Edmund Long in its report [22] criticized the fact the action costs are static and that the behavior of the planner could be improved further with dynamic costs 2 . I do believe on this point but with the use of context preconditions, and according to my results, dynamic cost may not modify the planner behavior that much if context preconditions are overused. Finally, my advice to restore the usefulness of action costs would have been to reduce the use of the context preconditions.

Planning Failures

As far as I know, planning failures in GOAP have never been mentioned. I thought it was important to mention them: they have appeared more than 74% for both OC + and UC configurations in my experiment (1). A planning failure occurs when the planner failed to link the goal state and the NPCs' current state, so it returns an empty plan. There are two possible reasons for it: (i) when all actions are pruned by context preconditions or (ii) when (at least) one precondition cannot be satisfied. The first reason is easy to understand and, Figure 2.5 highlights that it is almost all the time due to context preconditions. The second reason can be highlighted in F.E.A.R. with the action AttackFromView. This attack has AtN odeT ype = V iew as a precondition, which can only be satisfied when the NPC is on a node of the map labeled as View. So, either the NPC is already on such a node or a moving action such as GotoNodeOfType has to be performed. But the surroundings of an NPC can be without a View node due to map design, therefore in this situation, the precondition cannot be satisfied and the planner will fail to build a valid plan.

Conclusions

I analyzed in-game planning data from playing F.E.A.R. to evaluate the use of action costs in GOAP in order to provide an answer to the important game AI development problem of designing these action costs when starting a new game. I found that the behaviors of NPCs are the result of context preconditions: action costs have a negligible impact on NPC behaviors. Therefore if your implementation of GOAP closely follows that of F.E.A.R., action costs can be removed from this implementation in favor of context preconditions; this comes at the cost of some CPU cycles, however, and procedural precondition run time should not be overused but closely monitored to satisfy the time budget for GOAP.

I also analyzed how the heuristic for the A* search algorithm has been designed and used in GOAP: I explained that it actually fosters short action plans instead of cheap action plans. Video games are facing the major drawbacks of planning, pointed by D. Chapman [23], it is intractable in general. Game industries try to contain the intractability by restricting the number of calls per NPCs and per frame [3,5,24] and by restricting the size of the solution plans [21]. Despite these restrictions, planners in GOAP remain search algorithms like A * [15] or Djikstra [25] whose complexity are exponential with respect to the size of the solution plan, which, in return, does not help to improve GOAP or real-time planning system.

Eventually, even though the context preconditions in F.E.A.R. decrease the runtime performance of the GOAP planner, I strongly believe that this is the way to go. In fact, among the answers I get from asking Lead AI : "Do you use context preconditions?", are Troy Humphrey's answer (Highmoon Studio) who told us that these kind of preconditions are a bad because developers will put too many instructions inside, including heavy function like pathfinding (This is the case in some actions of F.E.A.R.), even though they are told not to. The result is that these instructions can be run several times while planning because there is no dedicated variable for them. This is the second part of Troy Humphrey's answer, the current form of the context preconditions is not optimal at all, they must be stored someway because sensing and planning are two different abstractions. Eventually, by considering that it exists a more efficient way to handle context preconditions and the fact that they overshadow the action costs in practice, I am convinced that action costs can be fully removed in favor of the context preconditions. The direct consequence will be that actions will become contextually post-unique.

I therefore made the following assumptions which I use in this thesis to produce tractable classes of problems and provide a correct and complete linear time algorithm to solve any instance of these classes.

Solution plans are totally ordered 4. There is not twice the same action inside a solution plan

The last assumption can be surprising at first glance. In fact, the size of the action plans are rather small, e.g. 1 or 2 actions at most in F.E.A.R. And it is slightly higher in some other video games using GOAP. And in these small plans, in particular in F.E.A.R., there is not twice the same action.

In the next Chapter, I introduce the Simplified Action Structure, a formalism created by [6] whose approach to counter the general intractability of planning was to apply restrictions on the action representation in order to find tractable subclasses of problems, and then to provide polynomial time algorithms to solve any instance of these problems. Then, because the assumptions of Proposition 2.4.1 are restrictions that are already implemented by game developers, the idea of the thesis is to find a tractable class of problems out of these assumptions and to provide an algorithm that can solve this new class of problems.

3

Simplified Action Structure

The Simplified Action Structure (SAS) was introduced by C. Bäckström [6] in the early 90's and it is a restricted version of the Action Structure formalism [26]. SAS is similar to the very well-known propositional formalism: Stanford Research Institute Problem Solver (STRIPS) [27], except that: (i) state variables are multi-valued and (ii) operators have two types of preconditions. (i) The multi-valuedness, which was said to be appealing for automatic control researchers, is also a feature of the planning system of GOAP (cf. Chapter 2). It is indeed more convenient than the add and delete lists of the STRIPS formalism, especially when using a programming language like C++ that allows the assignment of values. (ii) The two types of preconditions, which are latter described, are easily adaptable to the current representation of GOAP operators. The way these preconditions are described is algorithmically convenient because their description allows a simple definition of a Truth Criterion [23] which is a criterion useful to prove the correctness and the completeness of an algorithm. Eventually, GOAP operators are closer to those of SAS than those of STRIPS. D. Chapman proved that planning is intractable in general [23] and, unfortunately, the GOAP planning system is not an exception (cf. Chapter 2). C. Bäckström's approach to tackle the general intractability is to apply restrictions on SAS operators so as to create and study sub-classes of problems in the hope of finding usable and tractable ones [6,28]. Among these restrictions are the (P)ost-uniqueness and the (U)nariness [29] which directly concern 2 of my 4 assumptions (cf. Proposition 2.4.1). Without the addition of other restrictions, the class of problems SAS-PU is NP-Hard [30], however. The addition of the (S)ingle-valuedness and the (B)inariness restrictions to (PU) lead to SAS-PUBS and SAS + -PUS which are both tractable classes and which can both be solved by the polynomial time algorithms described in C. Bäckström's thesis [6].

In this Chapter, I recall how a planning problem is described with the SAS formalism and what is a minimal solution plan for a SAS problem instance. Then, I recall existing restrictions before focusing on SAS-PU and SAS-PUS, two consistent classes of problems for GOAP according to my assumptions.

SAS Planning Framework

In SAS, a planning problem is composed of a structure (Φ). Φ = (M, S, O) is a triplet made up of a set of multi-valued state variables (M), a set of states (S) and a set of SAS operators (O). The purpose of a SAS planner is to solve a planning problem instance ( ) by building an action plan (∆). = (Φ, s 0 , s ) is a triplet made up of a structure (Φ), an initial state (s 0 ) and a goal state (s ).

In this section, I first describe how to build a SAS structure, starting with the world state variables (M) and their associated domain. I then described how to design operators and what is the difference between an action and an operator. I then detail three different and existing SAS structures. Then, I explain what is an action plan and give further definitions useful to understand how a SAS problem instance can be solved.

World State Variables

In SAS, states of the world are described with state variables that can take several different values. A state is no more than a set of defined world state variables and it is used to describe an agent's situation at any time, or some goals. 1. M = {v 1 , ..., v m } is the set of state variables, where v i is the i th state variable.

D v i is the domain associated to v i and D

+ v i = D v i ∪ {u} is the extended domain, where "u" is the undefined value. S = { x 1 , x 2 , ..., x n | ∀x ∈ D v i } is the total-state space while S + = { x 1 , x 2 , ..., x n | ∀x ∈ D + v i } is the partial-state space. 3. Let s ∈ S + and v i ∈ M, s[v i ] is the value of v i in s.
4. s 0 , s are resp. the start and the goal states of SAS problem instances1 :

• s 0 , s ∈ S, with the SAS-structure [6, p.61].

• If s 0 , s ∈ S + , with the SAS + -structure [6, p.51-52].

• If s 0 ∈ S, s ∈ S + , with the SAS * -structure [28, p.42-43].

In order to illustrate how a video game world can be modeled with the SAS formalism, let introduce the NPCs of Figures 3.1 and 3.2. The first one represents the Horse Breeder's world: he can interact with haystacks, a bucket, a horse feeder, a source of water and a horse trough. I thus added the following state variables: v 0 for a haystack, v 1 for the bucket, v 2 to state the transfer of a certain amount of water ; associated with the following domain: • v 0 : HayStack, D v 0 = {0 : none, 1 : inHands, 2 : inF eeder}.

• v 1 : Bucket, D v 1 = {0 : none, 1 : inHands}. • v 2 : W ater, D v 2 = {0 : inSource, 1 : inBucket, 2 : inT rough}.
The second figure represents the Native hunter's world, where he has a hunting routine. I created the following state variables: v 3 represents the hunting routine, v 4 states the hunt readiness, v 5 and v 6 are two boolean variables that respectively check whether the Native hunter is rested and hydrated. These variables have the following domain:

• v 3 : HuntingRoutine, D v 3 = {0 : notStarted, 1 : onGoing, 2 : done} • v 4 : HuntingReadiness, D v 4 = {0 : notReady, 1 : ready} • v 5 : Energy, D v 5 = {0 : low, 1 : rested} • v 6 : T hirst, D v 6 = {0 : thirsty, 1 : hydrated} The sets M HB = {v 0 , v 1 , v 2 } and M NH = {v 3 , v 4 , v 5 , v 6 }
are respectively the set of state variables of the Horse Breeder and of the Native Hunter. Each of these two sets models the world for the corresponding type of NPC, focusing on what one wants or allows them to do. Of course, state variables can be combined differently: {v 0 , v 1 , v 2 , v 5 , v 6 } is a Horse Breeder that feels the thirst and tiredness; {v 0 , v 1 , v 2 , v 3 , v 4 , v 5 , v 6 } is a Native Horse Breeder and Hunter.

A state is written s

= v i = x | ∀v i ∈ M, x ∈ D v i
and it describes a particular situation. In video games, it is safe to say that, at any time, the state of an NPC is fully known, which implies we will only use the SAS-or SAS * -structures [6,28]. Let gives some state examples with the Horse Breeder:

• s = 0, 0, 0 , the trough, the feeder and the bucket are empty, and the Horse Breeder carries nothing.

• s = 2, 0, 2 , the trough and the feeder are filled and the bucket is dropped. • s = 1, 1, 1 , the Horse Breeder is carrying a haystack and a bucket full of water.

• s = 2, u, u , the feeder is filled and we do not know the state of neither the bucket nor the water travel.

I mainly use this state representation to define the current state of NPCs (s 0 ) and their goals (s ). The NPC's current state s 0 is updated by its sensors (ears, eyes, memory, collision boxes etc.) and by the successes of its action executions. Sensors update the current state in real-time while actions update it when their execution is over and succeeded. In case of action failure, the concerned state variables are obviously not modified by the action. The goals represent what NPCs want to accomplish according to their will, their belief, the context, etc. The goal determination, which is the Goal-Oriented system role in GOAP, is not studied in this thesis. I will define some of them in Chapter 5, however, where experiments and results are given.

Operators and Actions

Once the world is modeled, NPCs need operators to plan and actions to act in its environment. In fact, operators are instantiated by a SAS planner to add actions in the solution plan. Hence, an action is an instance of an operator. In other words, a SAS planner has operators as input and it return action plans as output. An operator can be instantiated as many times as needed by the planner, it results that a solution plan can have the exact same action several times.

Definition 3.1.2. An action is an instance of an operator, I use the following notations throughout the entire manuscript:

• The operator set is denoted: • The action set is denoted: A. An action of A is denoted: a.

O. An operator of O is denoted: o. O Pre Post Prevail v 0 , v 1 , v 2 Name of the operators/actions o 0 1, u, u 0, u, u u, u, u DropHaystack o 1 0, u, u 1, u, u u, 0, u TakeHaystack o 2 1, u, u 2, u, u u, u, u FillHorseFeeder o 3 u, 1, u u, 0, u u, u, u DropBucket o 4 u, 0, u u, 1, u 0, u, u PickUpBucket o 5 u, u, 0 u, u, 1 u, 1, u FillBucketWithWater o 6 u, u, 1 u, u, 2 u, 1, u FillHorseTrough v 0 : HayStack, D v 0 = {0 : none, 1 : inHands, 2 : inF eeder}. v 1 : Bucket, D v 1 = {0 : none, 1 : inHands}. v 2 : W ater, D v 2 = {0 : inSource, 1 : inBucket, 2 : inT rough}

• ope()

: A → O is a function s.t. a ∈ A, o ∈ O, ope(a) = o means a is an instance of o.
In C. Bäckström's thesis, an operator is an action type [6, p.48]. I have decided to use the more usual term, operator, which is also what C. Bäckström did with P. Jonsson in [28]. I also believe the term operator to be less confusing than the term action type.

Traditionally in planning, operators are described with preconditions and postconditions. With the SAS formalism, however, operators are represented with three different conditions: the pre-conditions, the post-conditions and the prevail-conditions. Actions being operator instances, they inherit this representation. The pre-and prevail-conditions are both preconditions. They both describe what must be satisfied before an action execution. The difference is that prevail-conditions hold values that will not be modified, whereas pre-conditions describe what will be modified. The pre-conditions are in fact linked with the post-conditions as the post-conditions describe the new values of the state variables affected by the action. Each of the three conditions define a partial-state or a total-state.

: O → S + Remark: Let v i ∈ M, pre(o)[v i ], post(o)[v i ], prv(o)[v i ]
are the value of v i in the pre-, post-and prevail-conditions respectively (cf. 3. Def 3.1.1).

Let illustrates the three types of conditions in SAS with the Horse Breeder whose operators are described in Table 3.12 . Let consider in particular the FillBucketWithWater operator, its execution will modify the W ater value from 0 to 1, i.e. a certain amount of water from the source (fountain) (0) is transferred in the bucket (1). Therefore, (v 0 = 0) is a pre-condition of FillBucketWithWater and (v 0 = 1) is a post-condition. To perform this action the NPC must have the bucket in his hands, and it must remain in it during the whole action execution. Hence, (v 1 = 1) is a prevail-condition of FillBucketWithWater.

SAS Planning Structures

Any SAS problem is composed of a structure, denoted Φ. A structure Φ = (M, S, O) is a triplet composed of a set of state variables M, a set of states S and a set of operators O. There exists three different structures, respectively denoted: SAS, SAS * and SAS + . These three structures all have the additional restrictions S1, S3, S4 and S5, but they differ with the restrictions S2, S6, S7 and S7(s 0 ). With dim(s) = {s[v i ] = u | v i ∈ M} being the set of defined state variables in s ∈ S + , the restriction S1 to S7(s 0 ) are described as followed: • S7: s 0 , s ∈ S.

• S1: For all o ∈ O, pre(o), post(o), prv(o) are consistent. • S2: For all o ∈ O, dim(pre(o)) ⊆ dim(post(o)).
• S7(s 0 ): s 0 ∈ S.

Restrictions S1 to S5, and S6, S7, were introduced in [6, p.52, p.61]. I denote S7(s 0 ) the restriction 1. of Definition 7.1 [28, p.42,43] as it is restriction S7 but only for s 0 . S7 and S7(s 0 ) could be rewritten differently but I wanted to keep the notation S7 for the sake of overall consistency, so that the readability between researches is consistent.

The restrictions S1, S3, S4 and S5 are applied to the three structures (SAS, SAS * and SAS + ) and they can be seen as "intuitive" restrictions. S1 is meaningless in this manuscript because it is about contradictory values k which we do not use in video games 3 . Therefore, S1 will always be respected in the following. S3 concerns the definitions of the postconditions and the prevail-conditions. The post-conditions describe what will change after the action execution while the prevail-conditions describe what must not change. So any defined state variable in the post-condition cannot be defined in the prevail-condition, and vice versa. A state variable cannot change its value while being asked to remain constant. S4 means that for each v i ∈ M, if an operator o ∈ O has its pre-condition defined on v i , its post-condition on v i cannot have the same defined value. An action cannot change a value from a state variable to the same value, if the value of a precondition must not change during the execution of an action, it is merely a prevail-condition. S5 is about operators having the same post-conditions. If two operators have the same post-conditions, then the pre-and prevail-conditions of one of the two must not be subsumed to the other one. If S5 is respected, then the structure is said to be parsimonious. As I'm interested in postunique operators, S5 will always be respected in my research because post-unique means that there are no two operators with the same post-conditions. Finally, and additionally to S1, S3, S4 and S5, applying the restrictions S2, S6, S7 or S7(s 0 ) highlight which structure is being used:

• SAS uses the additional restrictions S6 and S7.

• SAS * uses the additional restrictions S6 and S7(s 0 ).

• SAS + uses the additional restriction S2.

What differs between the different structures is the definition of the start and the goal states and their interaction with them. The SAS + -structure is the more general one. It allows the start and the goal to be partial-states. Hence, the restriction S2 in SAS + allows some operators to define a previously undefined state variable. That is for a given state variable, a pre-condition of an operator can be undefined and its post-condition defined. The SAS * -structure is a bit more restrictive as the start state must be totally defined. The SAS-structure, finally, is even more restrictive as the goal state must be totally defined as well. As a consequence for SAS and SAS * , the restriction S2 is no longer relevant and it is replaced with the restriction S6. The latter says that, if the post-conditions are defined for some state variables, then these state variables must be defined in the pre-conditions as well. Eventually, restrictions S2 and S6 both respect: dim(pre(o)) ⊆ dim(post(o)), which means that, in each structure, a post-condition cannot be undefined for a given state variable if it is defined in the pre-conditions. That is, an operator cannot "undefine" a defined state variable.

In order to evaluate partial-states in both SAS + and SAS * , the partial order was defined as followed [6]: Definition 3.1.4 (Partial order ).

• is a reflexive partial order on D + v i defined as: ∀p, q ∈ D + v i , p q ⇔ p = q or p = u.

• is a reflexive partial order over S + defined as:

∀s, s ∈ S + , (s s ⇔ (∀v i ∈ M, s[v i ] s [v i ])).
Let v i ∈ M and s, s ∈ S + , the partial order allows to compare a defined state variable s[v i ] = u with a possibly undefined one s

[v i ]. (s[v i ] s [v i ]) is equivalent to (s[v i ] = s [v i ])
, and the answer remains true if s [v i ] is undefined. For instance, let v i ∈ M, p ∈ D v i and s, s , s ∈ S + three possibles states of an intelligent agent, if

s[v i ] = s [v i ] = p and s [v i ] = u, then (s[v i ] s [v i ]) and (s[v i ] s [v i ]
) are both true.

SAS Planning Problem Instances

An instance of a SAS planning problem = (Φ, s 0 , s ) is composed of a structure Φ, an initial state s 0 ∈ S and a goal state s ∈ S.

is then solved by an action plan (∆). There is no cost function in SAS, so there is no "optimal" solution plan. There is minimal solution plan, however, and they are defined as follows: Definition 3.1.5 (Minimal Solution Plan). Let ≺ be a strict partial-order on a set of actions. ∆ = (A, ≺ A ) is a minimal solution plan solving if there is no other plan ∆ = (B, ≺ B ) solving such that |B| ≤ |A|.

In other words, a minimal solution plan solving is a solution plan with the minimum number of actions.

A Tractable Class of Problems: SAS-PUS

The SAS formalism is as expressive as Ground TWEAK, a propositional version of TWEAK, according to Corollary 28 [31], which implies unfortunately that, in general, SAS planning is intractable as well [23]. The approach of C. Bäckström to bypass this general intractability was to apply restrictions on SAS problems in order to create tractable classes of problems. Among those tractable classes of problems is SAS-PUS. It has three restrictions (P), (U) and (S), whose definitions is reminded below. Two of them, (P) and (U), which respectively stand for Post-unique and Unary, fit two of my assumptions from the conclusion of my study of GOAP (cf. Chapter 2). The following is organized as follows, I first remind the syntactical restrictions (P), (U), (B) and (S) and give useful existing definitions that I use in this manuscript. I then explain why I cannot use the class of problems SAS-PU for my research and give some results with the class SAS-PUS after reminding the theory behind.

Syntactical Restrictions

The restrictions Post-uniqueness (P), Unariness (U), Binary (B), Single-Valuedness (S) are syntactical [6]. They give rules on how to design operators (P, U, S), or on how many values a single state variable can have (B). They are defined as follows: Definition 3.2.1 (P,U,B ans S). Let Φ = (M, S, O) a SAS structure, Φ is:

∀v i ∈ M, |D v i | = 2. (S): Single-Valued iff, for some v i ∈ M, if it exists o, o ∈ O s.t. prv(o)[v i ] = u ∧ prv(o )[v i ] = u, then prv(o)[v i ] = prv(o )[v i ].
The single-valuedness means that it does not exist two operators with a defined prevailcondition on the same state variable that is not equal. Let consider the following SAS framework for the BlockWorld with 2 cubes: M = {AonB, BonA}, with D AonB = D BonA = {F, T}, i.e. each state variable only has two values: True or False. If AonB = F (resp. BonA = F), then it means A (resp. B) is on the table. We have the following operator set:

O pre post prv Explanation o 0 T, u F, u u, F A from B to Table o 1 F, u T, u u, F A from Table to B o 2 u, T u, F F, u B from A to Table o 3 u, F u, T F, u B from
v i ∈ M, for all o, o ∈ O st. prv(o)[v i ] = u, prv(o)[v i ] prv(o )[v i ]. The problem would not have been (S) if, for example, prv(o 0 )[v 1 ] = F ∧ prv(o 1 )[v 1 ] = T.
That is, a problem is not (S) if there exists at least 2 operators with a defined prevail-condition on the same state variable that is different.

C. Bäckström and P. Jonsson in [28] defined a useful set of values, called the requestable values, which is a set that stores, for each v i ∈ M and for unary operators, the values of D v i that are used as a prevail-condition on v i for some operators. Its definition is recalled below:

Definition 3.2.2. For each v i ∈ M, the set R O v i of requestable values for unary operators is defined as R O v i = {prv(o)[v i ] | o ∈ O} \ {u}.
To illustrate this set, let consider the Horse Breeder (Table 3.1) and the BlockWorld (Table 3.2) problems:

• Horse Breeder: R O v 0 = {0}, R O v 1 = {0, 1}, R O v 0 = ∅. • BlockWorld: R O v 0 = R O v 1 = {F}.
The requestable values are also useful to define the restriction (S). Indeed, a SAS problem is (S) iff |R O v i | ≤ 1 for each v i ∈ M. I use this set of values later in the manuscript.

SAS-PU and SAS-PUS

My approach to reduce the complexity of actual planning systems in video games is to restrict the planning domain, i.e. the input instead of the output. With respect to the assumptions made at the end of Chapter 2, the class of problems SAS-PU is the best match. Indeed, I made the assumptions that operators are (P) and (U) in commercial video games, and because the initial state of an NPC is always known, i.e. totally defined at each frame, the SAS-and SAS * -structures are more relevant than SAS + . SAS * authorizes uncertainty in the goal state, I therefore decided to focus on SAS first because SAS * can inherit the result of SAS while the reciprocal is false. Hence, the interest towards SAS-PU. But the class has been proven intractable (Theorem 6.14 [6]), unfortunately, there is therefore a necessity to add at least another restriction in order to have a tractable class of problems. This is feasible because, by adding the restriction (S), C. Bäckström proved that the resulting class of problems SAS-PUS is tractable. In addition, he has provided several versions of a correct and complete polynomial-time algorithm solving SAS-PUS problem instances. His algorithm has been proven correct and complete thanks to a truth criterion called IMP for Intended Minimal Plans. This truth criterion is based on the notion of chain of actions that is recalled below.

Definition 3.2.3 (Chain of actions "chain

v i "). Given a SAS-structure Φ = (M, S, O), two states s, t ∈ S and v i ∈ M, σ = chain v i (s[v i ], t[v i ]) = a 1 , ..., a n is a chain of actions over Φ from s[v i ] to t[v i ] iff either: 1. a) a k affects v i for k ∈ [[1; n]] b) pre(a 1 )[v i ] = s[v i ] c) post(a n )[v i ] = t[v i ] d) post(a k )[v i ] = pre(a k+1 )[v i ], for k ∈ [[1; n[ or, 2. σ is the empty chain v i = and s[v i ] = t[v i ].
Definition 3.2.4 (Minimal chain of actions "δ v i "). For each v i ∈ M, x, y ∈ D v i , x = y, I denote δ v i (x, y) the minimal chain from x to y. δ v i (x, y) is a minimal chain of actions from x to y iff there is no other chain v i (x, y) such that:

|chain v i (x, y)| < |δ v i (x, y)|.
For SAS-PU problems, SAS-PUS ones included, a chain of actions is therefore a linear sequence of actions that focus on the relation between the post-condition and the precondition for a given state variable. Then, minimal chains of actions have two interesting properties: there is not twice the same action in a minimal chain and it is unique. Lemma 3.2.1. Let x, y ∈ D v i , δ v i (x, y) does not have twice the same actions.

Proof. Let v i ∈ M, O(v i ) a set of (PU) operators affecting v i and A(v i ) a set of instances of the operators of O(v i ). A(v i ) is therefore a set of actions affecting v i .

Base case:

Let o 1 v i ∈ O(v i ) such that pre(o 1 v i ) = 0 and a 1 v i is an instance of o 1 v i . δ v i (0, 1) = a 1
v i is the minimal chain and only has one action.

Induction hypothesis: Let k ∈ D v i , {a 1 v i , ..., a k v i } ⊆ A(v i ) such that δ v i (0, k) = a 1 v i , .
.., a k v i is the minimal chain from 0 to k and there is not twice the same action in it, i.e. for each action a in δ v i (0, k), a is the only instance of ope(a) ∈ O(v i ).

Induction step:

Let l ∈ D v i , b ∈ A(v i ) such that pre(b)[v i ] = k ∧ post(b)[v i ] = l. If chain v i (0, l) = δ v i (0, k) ⊕ b is not the minimal chain from 0 to l, then ∃c ∈ δ v i (0, k) such that post(c) = l. By post-uniqueness, ope(c) = ope(b) = o l v i , and pre(c) = pre(b) = k, which implies: |chain v i (0, pre(c))| = |chain v i (0, k)| ≤ |δ v i (0, k)|.
It contradicts the hypothesis that δ v i (0, k) is the minimal chain. Therefore, δ v i (0, l) = δ v i (0, k) ⊕ b is the minimal chain from 0 to l and c ∈ δ v i (0, l) such that ope(c) = ope(b). This concludes the proof. Lemma 3.2.2 (Minimal chain uniqueness). Let v i ∈ M, x, y ∈ D v i such that x = y. δ v i (x, y) is the unique minimal chain between x and y.

Proof. Let v i ∈ M, x, y ∈ D v i and δ v i (x, y) and δ v i (x, y) two minimal chains between x and y. We have ,y) and it concludes the proof.

|δ v i (x, y)| = |δ v i (x, y)| = n. Lemma 3.2.1 and post-uniqueness imply that: ∀k ∈ [[1, n]], ope(δ v i (x, y)[k]) = ope(δ v i (x, y)[k]). Therefore, δ v i (x, y) ≡ δ v i (x
C. Bäckström relied on chains of actions to define the IMP. Given a SAS planning problem = (Φ, s 0 , s ), the IMP says that: a minimal solution plan ∆ = (A, ≺) is composed of chains such that, for each

v i ∈ M, chain v i (s 0 [v i ], s [v i ]) link s 0 [v i ] to s [v i ] ; Then, for each v i ∈ M, for each a ∈ A, if prv(a)[v i ] = q ∈ R O v i , then there exists δ v i from s 0 [v i ] to q and from q to s [v i ] such that δ v i (s 0 [v i ], q) ≺ a and a ≺ δ v i (q, s 0 [v i ]) ; Finally,
A is minimal wrt. the above and ≺ is a minimal strict partial order on A satisfying the above.

The notion of truth criterion was developed by D. Chapman [23, p.340] with his modal truth criterion (MTC). He proves that plans from TWEAK, his algorithm, are correct if and only if the MTC is respected. The MTC describes, for a given planning problem instance, how actions must at least be ordered to ensure a correct non-linear plan. To this end, Chapman orders two types of action, namely the establisher T and the clobberer C, in relation to a situation s where a proposition p has to be asserted. The establisher is the action that assert p to s, while the clobberer is an action that threatens (or clobbers) the assertion of p. In other words, a clobberer is an action that remove p from s. While the establisher must obviously be ordered before the situation s, Chapman suggested three possibilities for deblobbering: (1) Promotion: it consists in ordering the clobberer after s ; (2) Demotion with a White Knight: it consists in finding another establishing situation after the clobberer, called the white knight, and where p has to be asserted ; (3) Separation: it consists in avoiding the codesignation between the clobberer and the establisher. Based on this work, some other researchers [32,33] described that the situation s is in fact related to the precondition of an action, so the definition of a truth criterion can be more actionfocused by considering an action to assert instead of a situation. Then, É. Jacopin proved the "uselessness" of white knights [34]: a clobberer must in fact not be placed in between the establisher and the action to assert, it results that a clobberer has to be ordered before the establisher (Demotion) or after the action to assert (Promotion). Figure 3.4 presents a graphic parsing of PWEAK truth criterion, with PWEAK being É. Jacopin's planning algorithm: T is the establisher, C is the clobberer and A is the action to assert. IMP is not without similarities to what is described above. First of all, like D. Chapman's MTC, the IMP was used by C. Bäckström as a specification for plan generation algorithm. Then, in SAS, the action to assert, the establisher and the clobberer can be defined as follows: Definition 3.2.5. Let Φ = (M, S, O) a SAS-PU structure and A the set of actions that are instances of the operators of O. I denote a ∈ A the action to assert with a proposition p, T ∈ A the establisher and C ∈ A the clobberer. They are defined as follows:

1. The assertion of p either concerns the pre-condition of a: pre(a)[v i ] = p, or its prevail-condition on v i : prv(a)[v i ] = p.

The action T establishes p iff post(T)[v

i ] = p.

The action

C clobbers p iff pre(C)[v i ] = p.
Even though there are two types of preconditions in SAS: the pre-and prevail-conditions, it is just a case distinction in the truth criterion, i.e. the assertion of p concerns one or the other. The distinction between the pre-and prevail-condition in SAS is besides very convenient for a truth criterion on (PU) operators, especially thanks to the definition of the pre-condition which allows to define clobberers very easily. Indeed, by definition, the pre-condition of a (PU) operator in a SAS-structure is precisely the value of a state variable that will be changed by the instance of it, i.e. the action. In other words, each action clobbers what is defined in their pre-condition. Therefore, a clobberer in a SAS-PU problem is an action whose pre-condition is equal to the pre-or one of the prevail-conditions of another action. If a clobberer C clobbers the pre-condition of a, this is a clobbering situation that occurs in a chain of actions. But, thanks to post-uniqueness and unariness, there is one possible ordering in a chain of actions, i.e. there is no need for declobbering because actions are chained based on their post-condition and not based on their pre-condition. On the contrary, each defined prevail-condition of an action a ∈ A may have an establishing situation and a clobbering situation to handle. This was taken into account by the IMP with the two minimal chains

δ v i (s 0 [v i ], q) and δ v i (q, s [v i ]): for each v i ∈ M, q ∈ D v i such that prv(a)[v i ] = q, q
is the proposition to assert, hence the establisher of a is the minimal chain δ v i (s 0 [v i ], q) and the clobberer of a is the minimal chain δ v i (q, s [v i ]). More precisely with Definition 3.2.5, the establisher is the last action of δ v i (s 0 [v i ], q) and the clobberer is the first action of δ v i (q, s [v i ]). In the IMP, the declobbering technique only consists in promoting a before the clobberer: a ≺ C = F irst(δ v i (q, s [v i ])). The reason the IMP only requires the promotion technique is due to (S). For each v i ∈ M, we have |R O v i | ≤ 1 due to (S), let q ∈ R O v i be the single-value of D v i , Theorem 4.4 [6, p.76] indicates that a minimal solution plan ∆ for a SAS + -PUS planning problem can have at most two instances of the same operator, but it can be highlighted that the establisher of q in ∆ is never concerned by the double instantiation, i.e. there is a unique establisher of q in every minimal solution plan of a SAS-PUS problem. Therefore, it only exists one situation where the proposition q = prv(a)[v i ] is asserted. In a minimal solution plan of an SAS + -PUS problem, the clobberer must therefore be ordered after the action a to assert because once q has been clobbered, it does not exist another situation where q is asserted. It explains why, due to (S), the promotion technique is sufficient to declobber in C. Bäckström's IMP.

Eventually, like PWEAK truth criterion, the IMP does not use the separation technique. É. Jacopin [33] removed the separation technique of D. Chapman's MTC for PWEAK by considering that an action cannot be both an establisher and a clobberer. Due to the SAS structure restriction (S4), which says that an operator cannot have its post-condition equal to its pre-condition, an operator cannot both establish and clobber a proposition, so the separation technique is also not needed in the IMP.

The idea of the IMP is that: for each v i ∈ M, the single-value q ∈ R O v i is a unique subgoal on v i and, therefore, it has to be reached by the chain of actions on v i between the start (s 0 [v i ]) and the goal (s [v i ]). Let denote T the establisher of q (post(T )[v i ] = q) and C the clobberer (pre(C)[v i ] = q), the chain of actions chain

v i (s 0 [v i ], s [v i ]) has two possible forms: If T ∈ δ v i (s 0 [v i ], s [v i ]), then: chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], s [v i ]) (3.1) If T / ∈ δ v i (s 0 [v i ], s [v i ]), then T ∈ δ v i (s 0 [v i ], s 0 [v i ]) = such that: chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], q) ⊕ δ v i (q, s [v i ]) = δ v i (s 0 [v i ], q) ⊕ δ v i (q, s 0 [v i ]) ⊕ δ v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], s 0 [v i ]) ⊕ δ v i (s 0 [v i ], s [v i ]) (3.2)
In both equations, there is the minimal chain of actions

δ v i (s 0 [v i ], s [v i ]
) (possibly empty) and then, depending on whether q is established by the minimal chain between s 0 [v i ] and s [v i ], case 3.1 or case 3.2 should be considered. This is the planning strategy of C. Bäckström's algorithm so as to find the minimal solution plan ∆ = (A, R), with R the set of orders, while respecting the IMP:

Phase 1: For each v i ∈ M, build the minimal chain

δ v i (s 0 [v i ], s [v i ]). We have chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], s [v i ]
). Actions of these chains are added to A and their relations between their pre-and post-conditions are added to R.

Phase 2: For each a ∈ A, for each

v i ∈ M, (i) if prv(a)[v i ] = q = u and T, C ∈ chain v i (s 0 [v i ], s [v i ]), then a is ordered after T and before C. If q = s 0 [v i ], a is simply ordered before C. If q = goal[v i ],
a is simply ordered after T. The orders are added to R.

(ii) If T, C / ∈ chain v i (s 0 [v i ], s [v i ]
), then the minimal chains δ v i (s 0 [v i ], q) and δ v i (q, s 0 [v i ]) are built, the actions added to A and their relation added to R. The chain between s 0 [v i ] and s [v i ] is updated:

chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], q) ⊕ δ v i (q, s 0 [v i ]) ⊕ δ v i (s 0 [v i ], s [v i ]). Now, T, C ∈ chain v i (s 0 [v i ], s [v i ]
) and step 2.(i) can be executed.

Phase 3: Check whether the set of orders R is not irreflexive. If it is not irreflexive, then fail. The plan ∆ = (A, R) is returned otherwise.

There are three versions of the algorithm, with different worst-case time complexities: [35], this version performs a transitive closure to check the irreflexivity. The transitive closure is based on the Floyd-Warshall algorithm [36] whose worst-case time complexity is

1. O(n 3 • m 3 )
O(|V | 3 ), with V the vertex set of a graph. O(|V | 3 ) ≡ O(|A| 3 ) = O(n 3 • m 3 ).

O(n

Discussion

Given the fact P has the best worst-case time complexity, I chose to work with this one. I have implemented it in C++14 with default settings for Microsoft Visual Studio 2019.

The code was optimized with the use of references and pointers as much as possible to avoid unnecessary copy constructors and other underlying procedures that can burden the planning runtime performance, especially with the manipulation of actions that are instantiated from operators. I then tested P on scalable and realitic benchmarks to observe its runtime performance. The experiments and results are fully explained in Chapter 5.

It can be noticed, however, from Figure 5.18, that my implementation of P was able to provide a plan with a length of at most 10 to a hundred NPCs in realtime (< 1.67ms) for each of my realistic benchmarks. It represents on average one plan every 15µs. In F.E.A.R., the planner returns one plan in 43µs on average (cf. Figure 2.7), and in 33µs on average without the context preconditions. But plans in F.E.A.R. were of length 1 or 2, and at most 4. In Shadow of Mordors [3], developers bound the number of calls to 50 NPCs per frame and action plans are shorts, i.e. the same length as those of F.E.A.R. In other words, if developers make the effort to create problems of the class SAS + -PUS for their NPCs, P is capable of controlling of a hundred of them in realtime and with longer plans than current planning systems. The single-valuedness (S) is very restrictive [35], however. it is tricky to represent realistic problems with it and can sometimes be a difficult conundrum, if the representation is possible without concessions though. (S) does not allow problems where an action needs a room to be on and another the room to be off for instance. This restriction will only allow the room to be on (or off) as a prevail-condition. Removing (S) from SAS-PUS is equivalent to considering the instractable class of problems SAS-PU, however. The source of intractability is linked to exponentially sized minimal solution plans. It exists problems where the minimal solution plan requires an exponential number of operator instances, while with (S), minimal solution plans have at most two times the same instance it. One of my assumption from Chapter 2 is that action plans in video games have at most one occurrence of the same action, i.e. one instance per operator. Is it possible to only consider SAS-PU problems whose minimal solution plans have at most one instance per operator? Does it exist a restriction to SAS-structures whose resulting class of problems with (P) and (U) only have minimal solution plans with one occurrence of the same action?

4

Topological Planning

In Chapter 2, I defined four assumptions (Proposition 2.4.1) that should help reaching real-time planning. Among them are the restrictions (P) and (U), which, once combined with a SAS-structure, create the class of problems SAS-PU, proven NP-Hard [30]. In the previous Chapter, I recalled the restriction (S) which creates the tractable class of problems SAS-PUS with (P) and (U). (S) is very restrictive, however, plus it exists problems in this class where minimal solution plans have two occurrences of the same action which does not fit my fourth assumption.

The intractability of the class SAS-PU is due to exponentially sized minimal solution plans. I can quote the following observation from C. Bäckström [6, p. 175] at the end of his thesis:

One source of intractability for planning is that instances are allowed that require exponentially sized minimal solutions. [...] This is non trivial (to remove such instances), however. There is no single set of restrictions having this effect. [...] Although it is not sufficient to bound the plans for achieving tractability, it is an inevitable step to take.

What is missing from this observation is the reason why some minimal solutions are exponentially sized action plans. An action is an instance of an operator and, therefore, an exponentially sized minimal solution is an action plan where some operators have been instantiated an exponential number of times. Proof 6.14 [6, p. 138] is a good example. The proof is based on the Gray Code [37] represented as a SAS-PUB problem where each operator modifies a single bit. Let Φ = (M, S, O) be a structure of such a Gray Code problem, the instance = (Φ, s 0 = 0, 0, ..., 0 , s = 1, 0, ..., 0 ) leads to an exponentially sized minimal plan mainly because the operator that switches the least significant bit from 0 to 1 is at least instantiated 2 (|M|-2) times. 4.1: The SAS + -PUB 3-bits Gray Code. (a) lists the operators with their pre-, postand prevail-conditions ; (c) shows the minimal solution plan for the 3-bits Gray Code problem instance = (Φ, s 0 = 0, 0, 0 , s = 1, 0, 0 ). s 0 and s are respectively the start and the goal states ; (b) shows the succession of states between s 0 and s when ∆ is applied ;

+ v 0 0, u, u 1, u, u u, u, u o - v 0 1, u, u 0, u, u u, u, u o + v 1 u, 0, u u, 1, u 1, u, u o - v 1 u, 1, u u, 0, u 1, u, u o + v 2 u, u, 0 u, u, 1 0, 1, u o - v 2 u, u, 1 u, u, 0 0, 1, u (a) v 2 v 1 v 0 s 0 0 0 0 0 0 1 0 1 1 0 1 0 1 1 0 1 1 1 1 0 1 s 1 0 0 (b) (c) ∆ = a + v 0 , a + v 1 , a - v 0 , a + v 2 , a + v 0 , a - v 1 , a - v 0 . Table
a + v i (resp. a - v i ) is an instance of o + v i (resp. o - v i )
and it switches the bit v i from 0 to 1 (resp. from 1 to 0). affecting the same bit loop together. For example with the operators o + v 0 and o - v 0 , they loop together through their post-pre dependencies.

This chapter is organized as follows. In section 4.1, I provide prerequisites that will be used throughout the entire chapter. Operator and action graphs are then defined in section 4.2 and are useful to present the Υ 1 -Truth Criterion (section 4.3) and the new structural restriction C k with which I created new classes of problems (section 4.4). Eventually, I present TopoPlan, a correct and complete linear time algorithm that solves SAS-PUC 0 , SAS-PUC S 2 and SAS-PUC * 2 problem instances.

Prerequisites

In this section, I present several definitions that take advantage of the restrictions (P) and (U), and that are useful for what follows.

Remark: As actions are instances of operators, all the next definitions concerning operators are applicable to actions. I denote (PU) operators the operators restricted with (P) and (U). The restrictions (P) and (U) allow to identify a (PU) operator o with the pair (v i , p), where p is the postcondition of o on v i .

Definition 4.1.1 ((PU) Operator Identifier). I denote o p v i the unique (PU) operator such that:

v i ∈ M, p ∈ D v i , post(o p v i )[v i ] = p.
Definition 3.1.3 gives the original functions to represent pre-, post-and prevailconditions of SAS operators, i.e. these functions return a partial-state. Because I'm using unary operators, however, it is relevant to represent the pre-and post-condition of any unary operator o p v i as a function that return the value of the affected state variable v i instead. Table 4.2 (a) shows the Horse Breeder operator set with the new operator representation using the following definition.

• SAS or SAS * : pre(), post() : O(v i ) → D v i , • SAS + : pre() : O(v i ) → D + v i , post() : O(v i ) → D v i .
The function prv() does not change: prv()

: O → S + O Pre Post Prevail N pre N prv N clob Explanation o 0 v 0 v 0 = 1 v 0 = 0 u, u, u {o 1 v 0 } ∅ ∅ DropHaystack o 1 v 0 v 0 = 0 v 0 = 1 u, 0, u {o 0 v 0 } {o 0 v 1 } {o 1 v 1 } TakeHaystack o 2 v 0 v 0 = 1 v 0 = 2 u, u, u {o 1 v 0 } ∅ ∅ FillHorseFeeder o 0 v 1 v 1 = 1 v 1 = 0 u, u, u {o 1 v 1 } ∅ {o 1 v 2 , o 2 v 2 } DropBucket o 1 v 1 v 1 = 0 v 1 = 1 0, u, u {o 0 v 1 } {o 0 v 0 } {o 1 v 0 } PickUpBucket o 1 v 2 v 2 = 0 v 2 = 1 u, 1, u {o 0 v 2 } {o 1 v 1 } ∅ FillBucketWithWater o 2 v 2 v 2 = 1 v 2 = 2 u, 1, u {o 1 v 2 } {o 1 v 1 } ∅ FillHorseTrough (a) (b) (c)

The notation o p

v i carries the assigned state variable, namely

v i ∈ M, thus: post(o p v i )[v i ] ≡ post(o p v i ) and pre(o p v i )[v i ] ≡ pre(o p v i ). Hence Definition 4.1.2.
In practice, it is possible to index each operator o p v i with the formula i × n + p for instance, with n being the size of the biggest state variable domain. Such identification allows to create a hasing table, like in the algorithm P, to instantiate actions in constant time (O(1)) while planning. Definition 4.1.1 combined with Definition 3.2.5 allow to define two new types of operator sets. Indeed, the defined pre-or prevail-conditions of an operator determine its predecessors.

Definition 4.1.3 (Predecessors sets). For each o p

v i ∈ O, the predecessors of o p v i are defined as follows:

• N pre (o p v i ) = {o q v j | v j = v i ∧ q = pre(o p v i )}, the set of operators that establish the pre-condition of o p v i . N pre (a p v i ) is a singleton due to (P). • N prv (o p v i ) = {o q v j | v j = v i ∧ q = prv(o p v i )[v j ]}, the set of operators that establish the prevail-conditions of o p v i .
In an action plan, an action instance of these predecessors is an establisher of an action instance of o p v i . Table 4.2 (b) gives examples of predecessor sets. For instance with the operator FillHorseTrough (o 2 v 2 ), it requires water in the bucket (v 2 = 1), hence its predecessor via the pre-condition is

FillBucketWithWater (N pre (o 2 v 2 ) = o 1 v 2 ).
To fill the horse trough, the Horse Breeder also requires the bucket in his hands (v 1 = 1), therefore the predecessor of

o 2 v 2 via the prevail-condition is PickUpBucket (N prv (o 2 v 2 ) = o 1 v 1 ). Let o p v i , o q v j ∈ O, if N pre (o p v i ) = {o q v j }, i.e. v i = v j ∧ post(o q v j ) = pre(o p v i ), then there is a post-pre dependency between o q v j and o p v i : the post-condition of o q v j solves the pre-condition of o p v i . If N prv (o p v i ) = {o q v j }, i.e. v i = v j ∧post(a q v j ) = prv(a p v i )[v j ],
then there is a post-prevail dependency between o q v j and o p v i : the post-condition of o q v j solves the prevail-condition of o p v i on v j . Let consider the notations ≺ pre and ≺ prv to represent the two different dependencies. They are two strict partial-orders between two operators

o p v i , o q v j ∈ O such that (o q v j ≺ pre o p v i ) ≡ o q v j ∈ N pre (o p v i ) and (o q v j ≺ prv o p v i ) ≡ o q v j ∈ N prv (o p v i ). N pre (o p v i )
and N prv (o p v i ) can both be initialized during a pre-processing by looking after the defined pre-or prevail-conditions. The process takes no more than |M| steps per operator to complete as there are |M| state variables to browse and it takes a constant time to identify each predecessor via a hashing table using Definition 4.1.1.

Similar to the requestable values (cf. Definition 3.2.2) which, for unary operators, are values that are assigned to the prevail-condition of some other operators, I define the requestable (PU) operators. Due to restrictions (P) and (U), and with the identifiers (cf. Definition 4.1.1), a requestable value is in fact the post-condition of a unique (PU) operator.

Definition 4.1.4 (Requestable Operator RO). I define RO the set of requestable operators of O. These operators solve a prevail-condition of at least one other operator of

O. RO = {o | v i ∈ M, o ∈ O(v i ), post(o)[v i ] ∈ R O v i } Similar to O(v i ), A(v i ) or Cycle(v i ), I denote RO(v i ) the
= {o - v 0 , o + v 0 , o + v 1 }.
It results from Definition 3.1.2 and Definition 3.2.5 that it exists problem instances where an action instance of a requestable operator is an establisher of the prevail-condition of one or more other actions in the solution plan.

Because I am interested in the number of action instances of an operator in a solution plan, I introduce the following output restriction denoted Υ k .

Definition 4.1.5 (Output restriction Υ k ). Let k ∈ N, an action plan ∆ is Υ k iff:
• the number of distinct action instances of the same operator occurring in ∆ is at most k.

• it exists at least one operator that has k distinct action instances in ∆.

I will use this definition to characterize any sequence of actions such as action chains or minimal action plans. It results from this definition that the minimal solution plan of the instance of the 3-bits Gray Code presented

v i ∈ M, δ v i is Υ 1 . In addition, if a plan is said to be at least Υ k then it exists n ∈ N such that n ≥ k and the plan is Υ n .
I extend the definition of Υ k to problems. A SAS problem is at most Υ k iff it exists problem instances whose minimal solution plan is Υ k and iff it does not exist problem instances whose minimal solution plan is Υ n , ∀n > k. Then, a SAS problem is at least Υ k iff it exists problem instances whose minimal solution plan is Υ k and it may exist problem instances whose minimal solution plan is Υ n , ∀n > k. I add this definition or this characterization of SAS problems because I am interested in problems that are at most Υ 1 . In other words, I aim to discard any SAS problem that is at least Υ 2 .

Moreover, according to the introduction of this Chapter and Theorem 6.14 [6, p.138], the Gray Code problem with n-bits is at least Υ 2 (n-2) , as it exists problem instances where the least significant bit is switched (from 0 to 1) 2 (n-2) times. Hence, there are 2 (n-2) distinct action instances of the operator doing this switch. Concerning the class of problems SAS + -PUS, Theorem 4.4 [6, p.76] implies that these problems are at most Υ 2 .

Lemma 4.1.1. Let

= (Φ, s 0 , s ) a SAS problem instance such that s 0 = s . If is solvable, then there exists a non-empty minimal solution plan solving and it is at least Υ 1 .

Proof. The idea of the proof is to show that a Υ 0 plan is equivalent to an empty plan, and that an empty plan cannot solve a problem instance where the start is different from the goal state. Then, I justify the statement "and it is at least Υ 1 " by showing that it exists problem instances whose minimal solution plan is exactly Υ 1 .

Let denote ∆ the minimal solution plan of . If is solvable and ∆ is Υ 0 , then ∆ = . Indeed, Υ 0 means the number of distinct action instances of the same operator in ∆ is at most 0. And if ∆ = , then1 result(s 0 , ∆) = s 0 = s . Therefore, ∆ does not solve which is a contradiction. Therefore, ∆ cannot be Υ 0 , i.e. empty.

Let o an operator of a SAS problem and a its action instance, i.e. Proof. Every SAS problem has at least one operator and therefore, every SAS problem has at least the problem instance described in Proof 4.1.1 and whose minimal solution plan is Υ 1 .

Operator Graphs

Similar to the state transition graph and the domain transition graph [28], or causal graph [38,39], which consist in representing a planning problem as a graph, I introduce three new graphs that focus on operators and actions and their relations between them: two of them are operator graphs and represent the planning problem, the third one is an action graph and is a graphical representation of the solution plan. These operator-and action-focused graphs shed light on operators and actions instead of states, state values or state variables. Furthermore, the output of a SAS planner being action plans it is convenient to represent them as a graph to visualize the relations between the actions. As mentioned in the introduction of this Chapter, it exists (PU) operators that loop together and, as a result, for some problem instances, they can be instantiated an exponential number of times, thus creating exponentially sized plans. My approach to break this intractability is therefore to study these cycles so as to suggest a structural restriction that insures these operators are instantiated at most once in each minimal solution plan of each instance of a problem. A structural restriction is a concept that was introduced by C. Bäckström and P. Jonsson in [28]. They have used graphs to study the interaction between values of state variable domains and then suggested restrictions to avoid some unwilling planner behavior.

Operator Graph

The first graph I introduce is the Operator Graph and it is described as follows:

Definition 4.2.1 (Operator Graph). I denote G O = (O, E O ) the operator graph, with O the vertex set and E O the directed edge set. Let v i , v j ∈ M, E O = E pre (O) E prv (O)
, and:

• E pre (O) = {(o q v i , o p v i ) | o q v i , o p v i ∈ O, o q v i ∈ N pre (o p v i )}

is the set of directed edges between two (PU) operators of O having a post-pre dependency. Each post-pre dependency such that o

q v i ≺ pre o p v i is represented as a directed edge from o q v i to o p v i . • E prv (O) = {(o q v j , o p v i ) | o q v j , o p v i ∈ O, o q v j ∈ N prv (o p v i )}
is the set of directed edges between two (PU) operators having a post-prevail dependency. Each post-prevail dependency such that

o q v j ≺ prv o p v i is represented as a directed edge from o q v j to o p v i .
An operator graph shows where the planner will navigate to search for a plan and gives a hint on how actions will be connected to each other for a given problem instance. In the introduction of this Chapter, I mentioned that only operators affecting the same state variable and looping between each other can be instantiated several times. In the Horse Breeder example, the operators of the pairs PickUpBucket/DropBucket and DropHaystack/TakeHaystack loop between each other and can be instantiated several times by the planner. For instance, if the horse breeder wants to work out with his haystacks, he can repeatedly take and drop them. σ = a 1

Domain Operator Graph

The following definition concerns the Domain Operator Graph, a useful graph to study chains of actions (cf. Definition 3.2.3).

2. G O (v i ) has |D v i | operators.
The property (1.) of Definition 4.2.2 is kind of a structural restriction. Although it may exist problems with disconnected domain operator graphs, each connected parts are weakly connected. Plus, to solve a problem instance, a SAS planner will only navigate through one of the connected part to find a solution. Let consider the following SAS planning framework,

M = {v 0 }, D v 0 = {0, 1, 2, 3} and O = {o 1 v 0 , o 3 v 0 } such that pre(o 1 v 0 ) = 0 ∧ post(o 1 v 0 ) = 1 and pre(o 3 v 0 ) = 2 ∧ post(o 3 v 0 ) = 3.
If the problem instance is made up of s 0 [v 0 ] = 0 and s [v 0 ] = 3, then it is not solvable because o 1 v 0 and o 3 v 0 are disjoint and, therefore, it does not exist a chain of actions between s 0 [v 0 ] = 0 and s [v 0 ] = 3. No matter the problem, the instances of o 1 v 0 and o 3 v 0 will never end up in the same chain of actions. The idea behind the property (1.) is therefore to say that, in such cases, the disjointed parts can be considered as different domain operator graphs, even though the operators affect the same state variable. It does not matter as their action instances will never end up in the same chain of actions, nor in the same action plan.

The property (2.) is directly linked to Definition 4.1.1. We can identify exactly

|D v i | operators, hence G O (v i ) has |D v i | operators.
In fact, the post-unicity implies that there are at most |D v i | operators, i.e. a unique operator for each value of the domain v i . The weakly connected property implies that there are at least

|D v i | -1 operators. Indeed, consider the domain transition graph G v i = (D v i , T v i ) [28, p.10], with T v i = {(pre(o), o, post(o)) | o ∈ O(v i ), pre(o), post(o) ∈ D v i } the set of transition between two values of the domain of v i , in order to connect each value of D v i we need at least |D v i | -1 arcs in T v i .
As an arc of T v i is merely an operator, it results that there are at least

|D v i | -1 existing operators in G O (v i ). Therefore, there may exist a unique x ∈ D v i such that o x v i / ∈ O(v i ), i.e.
it exists an unique operator that can be identified but that does not exist in the operator set. I call this missing operator the ghost operator. Proof 4.2.2 gives more details on why there are at least The Horse Breeder has a ghost operator in

|D v i | -1 operators with a defined pre-condition in G O (v i ). Definition 4.2.3 (Ghost operator). Let v i ∈ M, x ∈ D v i . The ghost operator 2 is an o v 1 ,- o v 1 ,+ o v 0 ,- o v 0 ,+ o v 2 ,+ o v 2 ,- (a
G O (v 2 ), named Ghost V2, whose identifier is o 0 v 2 (cf. Definition 4.2.

3).

operator that can be identified with the pair

(v i , x) according to Definition 4.1.1 but that does not exist in O, i.e. o x v i / ∈ O. The pre-condition of o x v i is unknown, therefore: pre(o x v i ) = ∅.
In the Horse Breeder problem, there is a ghost operator, named Ghost V2. It is represented in Figure 4.2 with a dashed node and a dashed directed edge to FillBucketWithWater. In fact, (v 2 = 0) states the water is in a source and, in my Horse Breeder model, no action can fill the source as I consider the filling to be an external event. Therefore, identifying (v 2 = 0) as an operator results in a ghost operator, i.e. an operator that can be identified but that does not exist.

Lemma 4.2.1 (Ghost Operator Uniqueness). For each v

i ∈ M, if there is a ghost operator in G O (v i ), then it is unique. Proof. Direct from the fact a ghost operator, if it exists, is a missing operator in G O (v i ). By definition of G O (v i ), there is at least |D v i | -1 defined operator. The last one, if not defined, is the ghost operator and is unique in G O (v i ).
The domain operator graphs of (PU) operators have two specific characteristics:

Lemma 4.2.2. For each v i ∈ M: 1. G O (v i ) is a directed tree if there is a ghost operator.

G O (v i ) has a unique cycle otherwise.

the one without (O). But it burdens the notation and does not improve the understanding. Therefore I consider the ghost operators to be part of O. A ghost operator cannot be instantiated.

Proof. The two points are proven by recursion. Let Φ = (M, S, O) a SAS-structure, v i ∈ M a state variable with D v i its domain, and

G O (v i ) |Dv i | = (O(v i ), E pre (O(v i ))
) the associated domain operator graph.

Base case:

Let D v i = {x 1 , x 2 },
(o 2 ) = x 2 . We have O(v i ) = {o 1 , o 2 } Case 1: There is a ghost operator in G O (v i ) 2 .
Let consider o 2 to be the ghost operator. Therefore, o 1 is not a ghost operator, and due to restriction (S4) and (S6) from the SASstructure (cf. subsection 3.

1.3), pre(o 1 ) = post(o 1 ) = u and pre(o 1 ) ∈ D v i \ {x 1 }, that is: pre(o 1 ) = x 2 . It implies that, o 2 ≺ pre o 1 , that is G O (v i ) 2 has 2 nodes and 1 directed edge. Therefore, G O (v i ) 2 is
a directed tree and the root node o 2 is the ghost operator. If o 1 and o 2 are swapped in the proof, the reasoning is equivalent.

Case 2: There is no ghost operator in G O (v i ). Therefore, due to restriction (S4) and

(S6), pre(o 1 ) = post(o 1 ) = u and pre(o 1 ) ∈ D v i \ {x 1 } and pre(o 2 ) = post(o 2 ) = u and pre(o 2 ) ∈ D v i \ {x 2 }. That is pre(o 1 ) = x 2 and pre(o 2 ) = x 1 , which implies o 2 ≺ pre o 1 and o 1 ≺ pre o 2 .
In other words, o 1 and o 2 are looping together, i.e. they create a cycle in G O (v i ) 2 and the cycle is unique.

Induction hypothesis:

(D v i ) n = {x 1 , ..., x n } and (O(v i )) n = {o 1 , ..., o n } such that ∀k ∈ [[1, n]], post(o k ) = x k . Case 1: (G O (v i ))
n is a directed tree and the root node is the ghost operator. The ghost operator is denoted o j and post(o j ) = x j .

Case 2: There is no ghost operator and (G O (v i )) n has a unique cycle.

Induction step: Let (D v i ) n+1 = {x 1 , ..., x n+1 } and (O(v i )) n+1 = {o 1 , ..., o n+1 } such that ∀k ∈ [[1, n + 1]], post(o k ) = x k .
Case 1, there are two possibilities:

• o n+1 is a ghost operator, it means o n+1 is disconnected from (G O (v i )) n . Indeed, post(o n+1 ) = x n+1 / ∈ (D v i
) n and its pre-condition is not set. Hence, o n+1 does not precedes nor succeeds any operator of (G O (v i )) n . So, o n+1 cannot be a ghost operator, otherwise (G O (v i )) n+1 is not weakly connected which contradicts the definition of the domain operator graph.

• o n+1 is not a ghost operator, and due to restriction (S4) and (S6), post

(o n+1 ) = pre(o n+1 ) = u, therefore pre(o n+1 ) ∈ (D v i ) n+1 \ {x n+1 }, that is pre(o n+1 ) ∈ (D v i ) n . It implies o n+1 is preceded by one operator in (G O (v i )) n . Therefore, this adds a new node o n+1 and a new directed edge from (G O (v i )) n to o n+1 , it implies (G O (v i )) n+1 has n + 1 nodes and n directed edges. (G O (v i ))
n+1 is therefore a directed tree and the root node o j is the only ghost operator. It concludes point 1. of the Lemma: If there is a ghost operator, G O (v i ) is a directed tree.

Case 2, there are two possibilities:

• If o n+1 is a ghost operator, then (G O (v i )) n+1 is not weakly connected because o n+1 does not connect (G O (v i )) n . Indeed, post(o n+1 ) = x n+1 / ∈ (D v i ) n and all the operators of (G O (v i )) n have their pre-condition defined in (D v i ) n .
It contradicts the definition of a domain operator graph, so o n+1 cannot be a ghost operator.

• If o n+1 is not a ghost operator, then due to restriction (S4) and (S6), pre

(o n+1 ) ∈ (D v i ) n+1 \{x n+1 }, that is pre(o n+1 ) ∈ (D v i ) n ,

Remark:

The restriction (S4) (cf. subsection 3.1.3) says that the post-condition and the pre-condition of an operator cannot be equal, it implies domain operator graphs have no loops3 . Figure 4.2b presents the domain operator graphs of the Horse Breeder (cf. Table 4

.2). G O (v 0 ) and G O (v 1
) have a unique cycle and no ghost operator while G O (v 2 ) is a directed tree with a ghost operator and a single branch. In Appendix B, I provide Mathematica functions to produce all possible shapes of a domain operator graph with (PU) operators with respect to the number of values in the domain. Let v i ∈ M, I also provide the result of these function for [2,5]]. I was not able to go higher due to memory and time limitations. Then, it can be noticed that the shapes of G O (v i ) as a function of |D v i | is equivalent to the graphs presented under the OEIS reference "A002861", whose integer sequence name is "Number of connected functions (or mapping patterns) on n unlabeled points, or number of rings and branches with n edges".

|D v i | ∈ [
I previously stated that what cause the intractability of the class of problems SAS-PU are the operators that can be instantiated more than once, that is, operators that are inside a cycle. It is therefore relevant to study the cycle highlighted by (2.) Lemma 4.2.2 because this is exactly the starting point of the intractability of the class of problems SAS-PU. To this end, I introduce a new set of operators that, for each v i ∈ M, stores operators looping together in the unique cycle of

G O (v i ). Definition 4.2.4. Let v i ∈ M, I denote Cycle(v i ) the set of operators that are inside the unique cycle of G O (v i ). If G O (v i ) is a directed tree, then Cycle(v i ) = ∅. Otherwise, Cycle(v i ) = ∅.
Let consider the Horse Breeder example to illustrate this new notation:

Cycle(v 0 ) = {DropHaystack, TakeHaystack}, Cycle(v 1 ) = {DropBucket, PickUpBucket}, Cycle(v 2 ) = ∅.
Similarly, in the 3-bits Gray Code example, there is for each

v i ∈ {v 0 , v 1 , v 2 }: Cycle(v i ) = {o - v i , o + v i }.
= (Φ GC , s 0 = 0, 0, 0 , s = 1, 0, 0 ). o + v 0 and o + v 0 are instantiated twice: ope(a v 0 ,+ ) = ope(a v 0 ,+ ) = o + v 0 and ope(a v 0 ,-) = ope(a v 0 ,-) = o - v 0 .
The blue edges are the post-pre dependencies and the green edges are the post-prv dependencies. The red ones are orders to handle clobberers (cf. Definition 3.2.5).

Action Graph

The action graph is the graphical representation of an action plan. In particular, the graphical representation of the minimal action plan ∆ is described as follows: Definition 4.2.5 (Minimal Action Graph). Let ∆ = A, ≺ a minimal action plan, I denote G ∆ = (A, E A ) the minimal action graph, with A the set of vertices and E A the set of directed edges, such that:

• G ∆ ≡ ∆, G ∆ is the graphical representation of ∆. • E A = {(a, a ) | a, a ∈ A ∧ a ≺ a }.
The topological sort of G ∆ is equivalent to a totally-ordered minimal action plan. Figure 4.3 gives an example of an action graph: (4.3a) represents the non-linear minimal action plan solving the 3-bits Gray Code problem described in the introduction of this Chapter, and (4.3b) is the topological sort of it. The blue edges represent the post-pre dependencies and the green edges represent the post-prv dependencies. With Chapman's terms (cf. Definition 3.2.5), the tail of a blue or green directed edge is an establisher and the head is the action to assert. These two types of edges represent the establishment process. The red type of edges, however, represents the declobbering process with the promotion technique (cf. subsection 4.3.2). The head of a red edge is a clobberer and the tail is the action threatened by the clobberer. Without these red directed edges, it exist topological sorts that are not correct.

Discussion

It should be noticed that the operator graph and the domain operator graph for (PU) operators are respectively the state transition graph and the P-Graph of [28] except that my graphs focus on operators instead of state variable values. That is, the arcs of their graphs are the nodes of mine. Definitions 7.3 and 7.4 [28, p.44] say that the P-Graph is composed of a "center" which is a set of values either empty or it forms a simple cycle where vertex has incoming and outgoing degree of 1. This set of values, once their associated operator identified (Definition 4.1.1), is equivalent to the set of operators: v i ∈ M, Cycle(v i ). They did not prove, however, why the cycle is unique. This is now proved with Lemma 4.2.2. Definition 7. 4 gives another interesting description of the P-Graph and which can be transposed to the domain operator graph. Each element of the cycle is a root node of a (possibly empty) directed tree. Let takes G O (v 0 ) from the Horse Breeder (cf. Figure 4.2b), the center is: Cycle(v 0 ) = {DropHaystack, TakeHaystack}. DropHaystack is a root node of an empty tree while TakeHaystack is a root node of a directed tree composed of a single branch that connects TakeHaystack to FillHorseFeeder.

Truth Criterion

A minimal solution plan for a SAS-PU problem instance is composed of several chains of actions between the start and the goal state, and more precisely it is composed of a chain of action (possibly empty) for each state variable. This assertion from C. Bäckström's IMP holds when (S) is relaxed. His truth criterion, however, is adapted for SAS + -PUS problems that are at most Υ 2 . In this section, I aim to provide a truth criterion for SAS-PU problem instances whose minimal solution plans are Υ 1 . The section is organised as follows: I present the general form of a chain of actions, i.e. the possible forms of a chain of actions that is Υ k . Based on that, I present the possible forms when k = 1. Out of these forms, I present several clobbering situations that must be handle in order to define a truth criterion for SAS-PU problem instances whose minimal solution plan is Υ 1 .

General Form of an Action Chain

We can observe from the 3-bits Gray Code instance = (M, O, s 0 = 0, 0, 0 , s = 1, 0, 0 ) (cf. Figure 4.1) that the chain of actions on v 0 of the solution plan can be written:

chain v 0 (s 0 [v 0 ], s [v 0 ]) = chain v 0 (0, 0) = a + v 0 , a - v 0 , a + v 0 , a - v 0 = δ v 0 (s 0 [v 0 ], s 0 [v 0 ]) = a + v 0 ,a - v 0 ⊕ δ v 0 (s 0 [v 0 ], s 0 [v 0 ]) = a + v 0 ,a - v 0
If we extend the instance of Figure 4.1 to the n-bits Gray Code ( = (M, O, s 0 = 0, 0, ..., 0 , s = 1, 0, ..., 0 )), then the chain of actions of the least significant bit in the solution plan looks like:

chain v 0 (s 0 [v 0 ], s [v 0 ]) = δ v 0 (s 0 [v 0 ], s 0 [v 0 ]) ⊕ ... ⊕ δ v 0 (s 0 [v 0 ], s 0 [v 0 ]) O vi,x O vi,y O vi,z o 1 o 2 o 3 Figure 4.4: A generic Domain Operator Graph. Let v i ∈ M, o x v i , o y v i , o 1 , o 2 ∈ Cycle(v i )
, o 1 and o y v i are the root nodes of two different directed trees. The dashed directed edges represent possible hidden nodes, i.e. it is a choice of representation to illustrate that paths between nodes can be longer. The dotted directed trees whose root nodes are respectively o y v i and o 1 are here as an example to illustrate a generic shape of a domain operator graph.

Based on section 4.2, a domain operator graph is composed of a unique cycle (cf. Lemma 4.2.2) and each node of this cycle is the root node of a, possibly empty, directed tree. Figure 4.4 gives a generic representation of a domain operator graph. Since action chains are composed of actions instantiated from operators affecting the same state variable, these domain operator graphs help to understand how the size of a chain grows. Let k ∈ N, v i ∈ M, x, y, z ∈ D v i , based on Definition 4.1.5 and the study of the domain operator graphs (cf. section 4.2), a chain of actions is Υ k iff:

chain v i (x, z) = δ v i (x, x) ⊕ ... ⊕ δ v i (x, x) k-1 times ⊕δ v i (x, y) ⊕ δ v i (y, z) ∧ ∀a ∈ δ v i (x, x), ope(a) ∈ Cycle(v i ) ∧ ∀b ∈ δ v i (x, y), ope(b) ∈ Cycle(v i ) ∧ ∀c ∈ δ v i (y, z), ope(c) / ∈ Cycle(v i ) (4.1)
Let consider Figure 4.4 to understand why chain

v i (x, z) is Υ k . Let a x v i , a y v i , a z v i ,
δ v i (x, x) = a 1 , ..., a y v i , a 2 , ..., a x v i = δ v i (x, y) ⊕ δ v i (y, x) ∧ δ v i (x, y) = a 1 , ..., a y v i ∧ δ v i (y, z) = a 3 , ..., a z v i chain v i (x, z
) is therefore Υ k because the actions a 1 , ..., a y v i are instantiated k times: (k -1) times in δ v i (x, x) and once in δ v i (x, y). The chain δ v i (x, x) can be seen as the unfolding of Cycle(v i ) from x to x, while δ v i (x, y) is an unfolding from x to y. The chain δ v i (y, z) represents an external path to the cycle, from the post condition of a root node

(y = post(o y v i )) to the goal (s [v i ] = post(o z v i ) = z).
This external chain is obviously Υ 1 . In practice, what causes the multiple passage in the graph cycle are the requests of the requestable values that are satisfied by operators belonging to the cycle. If there is an alternation of appearance in the solution plan for instance: one value is requested, then another, then the first again, etc. That will cause the planner to create chains with several occurrences of the same actions. A typical example of this phenomena is the Gray Code: the logical state of each binary digit switch from 0 to 1 and from 1 to 0 constantly. Another good example is given in Appendix C.4c, the "odd" operators (o 1 , o 3 , o 5 ) have a positive prevail-condition on v i while the "even" operators (o 2 , o 4 , o 6 ) have a negative one. The solution plan of the appendix problem instance is composed of the sequence of actions a 1 , a 2 , ..., a 6 , so Cycle(v i ) = {o - v i , o + v i } has to be unfolded three times in order to produce a chain of actions that satisfies the prevail-condition on v i of each action of the sequence by alternation. Another possibility is that the requestable values are required by the solution plan in an inverse order to that of the operators that satisfy them. That will also cause the planner to create chains of actions with many occurrences of the same actions. It can also explain why chains of actions are at most Υ 2 when the restriction (S) is applied. There is at most one requestable value per state variable, so the alternation of appearance or the appearance in an inverse order have no meaning since there is only one requestable value per state variable domain. This value only needs to be reached once, at most, and there is no other requestable value that will require the planner to create a chain to reach it, which, in return, also prevents the planner from creating another chain of actions to reach the already reached once requestable value. Lastly, the Gray Code is a good example to illustrate that two requestable operators looping together are sufficient to create intractable problems.

Υ 1 Truth Criterion

Based on the notion of truth criterion described in subsection 3.2.2 and the adaptation of the terms "establisher", "clobberer" and "action to assert" for SAS-PU problems given by Definition 3.2.5, I neatly described in this subsection all the situations to handle for SAS-PU problems whose minimal solution plan is Υ 1 . I especially describe situations that were not possible with the restriction (S). Firstly, based on the previous subsection, chains of actions are Υ 1 iff they respect the following Lemma:

Lemma 4.3.1. Let v i ∈ M, x, y, z ∈ D v i , o x v i , o y v i ∈ Cycle(v i ) such that o y v i is a root node of a directed tree containing o z v i ∈ Cycle(v i ). chain v i (x, z
), a chain of actions between x and z, is Υ 1 iff:

chain v i (x, z) = δ v i (x, y) ⊕ δ v i (y, z) ∧ ∀a ∈ δ v i (x, y), ope(a) ∈ Cycle(v i ) ∧ ∀b ∈ δ v i (y, z), ope(b) / ∈ Cycle(v i ) (4.2)
Proof. Direct from the equation (4.1), by taking into account that k = 1.

Depending on the relation between x, y and z, a Υ 1 chain of actions will have the possible forms described in Table 4.3.

chain v i (x, z) = Description x = y y = z δ v i (x, y) ⊕ δ v i (y, z) ∀a ∈ δ v i (x, y), ope(a) ∈ Cycle(v i ) (a) ∧ ∀b ∈ δ v i (y, z), ope(b) / ∈ Cycle(v i ) y = z δ v i (x, y) ⊕ ∀a ∈ δ v i (x, y), ope(a) ∈ Cycle(v i ) (b) x = y y = z δ v i (y, y) ⊕ δ v i (y, z) ∀a ∈ δ v i (y, y), ope(a) ∈ Cycle(v i ) (c) ∧ ∀b ∈ δ v i (y, z), ope(b) / ∈ Cycle(v i ) ⊕ δ v i (y, z) ∀b ∈ δ v i (y, z), ope(a) / ∈ Cycle(v i ) (d) y = z δ v i (y, y) ⊕ ∀a ∈ δ v i (y, y), ope(a) ∈ Cycle(v i ) (e) (f) Table 4.3: Let v i ∈ M, x, y, z ∈ D v i .
The possible forms of a Υ 1 action chain depending on the relations between x, y and z. x is the start value, z the goal one and y the root one.

o x v i , o y v i ∈ Cycle(v i ) and o z v i / ∈ Cycle(v i ). Let consider a x v i , a y v i , a z v i ,
(a) chain v i (x, z) = δ v i (x, y) ⊕ δ v i (y, z) = a 1 , ..., a y v i ⊕ a 3 , ..., a z v i = δ v i (x, z) (b) chain v i (x, y) = δ v i (x, y) = a 1 , ..., a y v i (c) chain v i (y, z) = δ v i (y, y) ⊕ δ v i (y, z) = a 2 , ..., a x v i , a 1 , ..., a y v i ⊕ a 3 , ..., a z v i (d) chain v i (y, z) = δ v i (y, z) = a 3 , ..., a z v i (e) chain v i (y, y) = δ v i (y, y) = a 2 , ..., a x v i , a 1 , ..., a y v i (f) chain v i (y, y) = (c
) and (e) are chains that both fully unfold the cycle, i.e. they unfold the cycle from y to y. (a) and (b), on the contrary, partially unfold it, from x to y. Finally, (d) is an external path. With s 0 [v i ] and s [v i ] as parameters for the chain of actions, and based on the equations (a) to (e) (the equation (f) is trivial), there are three cases to handle in order to define a proper truth criterion for Υ 1 solution plan in SAS:

1. σ 1 = chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], s [v i ]) (which represents the chains (a), (b) and (d)) 2. σ 2 = chain v i (s 0 [v i ], s 0 [v i ]) = δ v i (s 0 [v i ], s 0 [v i ]) (which represents (e)) 3. σ 3 = chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], s 0 [v i ])⊕δ v i (s 0 [v i ], s [v i ]) (which represents (c)).
We have

σ 3 = σ 2 ⊕ σ 1 .
Let v i ∈ M, I denote q ∈ R O v i the proposition to assert and a ∈ A the action to assert such that prv(a)[v i ] = q. Similar to C. Bäckström's IMP, and based on Definition 3.2.5, the establisher of q, that I henceforth denote T, and its clobberer, that I henceforth denote C, both affect the state variable v i such that:

post(T)[v i ] = q and pre(C)[v i ] = q.
So it means that T and C are in the same chain of actions between s 0 [v i ] and s

[v i ]: T, C ∈ chain v i (s 0 [v i ], s [v i ])
. Therefore, to define any promotion or demotion technique, it is relevant to study the chains of actions of an action plan.

Case (1):

σ 1 = chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], s [v i ]). For each T ∈ δ v i (s 0 [v i ], s [v i ]
), T is post-unique due to Υ 1 and due to the fact operators are post-unique. Each value

q between s 0 [v i ] and s [v i ] is therefore established once, either by s 0 [v i ], or by T ∈ δ v i (s 0 [v i ], s [v i ]). By definition of the clobberer, q is clobbered once except when q = s [v i ],
where there is no clobberer. There are three possibilities:

• If q = s 0 [v i ] ∧ q = s [v i ],
a is established by T and promoted before C:

∃T, C ∈ chain v i (s 0 [v i ], s [v i ]) s.t. T ≺ C, (T ≺ a ∧ a ≺ C) (4.3) • If q = s 0 [v i ],
a is established by the start and promoted before C which is the first action of the minimal chain:

C = F irst(δ v i (s 0 [v i ], s [v i ])), a ≺ C (4.4) • If q = s [v i ],
a is established by T which is the last action of the minimal chain:

T = Last(δ v i (s 0 [v i ], s [v i ])), T ≺ a (4.5) Case (2): σ 2 = chain v i (s 0 [v i ], s 0 [v i ]) = δ v i (s 0 [v i ], s 0 [v i ]). If q = s 0 [v i ]
then q is established twice: once with the start and once with T s.

t. post(T )[v i ] = s [v i ] = s 0 [v i ]. That is, q is established by the start or by Last(δ v i (s 0 [v i ], s [v i ])). Then, the proposition q is clobbered once with C = F irst(δ v i (s 0 [v i ], s [v i ])). So, if q = s 0 [v i ],
there is two possibilities to assert the action a:

• The action a is established by the start and promoted before C which is the first action of the minimal chain:

C = F irst(δ v i (s 0 [v i ], s [v i ])), a ≺ C (4.6)
• The action a is established by T, which is the last action of the chain, and demoted after C:

∧ T = Last(δ v i (s 0 [v i ], s [v i ]), T ≺ a C = F irst(δ v i (s 0 [v i ], s [v i ]), C ≺ T (4.7)
It is worth noticing that:

δ v i (s 0 [v i ], s 0 [v i ]) = C, .
.., T . Therefore, C ≺ T is redundant because C is already ordered before T in the minimal chain of actions. Ordering a after T is therefore sufficient in these case as the demotion technique is spontaneously achieved by the building of

δ v i (s 0 [v i ], s 0 [v i ]). If q = s 0 [v i ],
then the assertion of the action a follows the rule given by the equation (4.3).

Case (3):

σ 3 = chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], s 0 [v i ]) ⊕ δ v i (s 0 [v i ], s [v i ]). Similar to case (2), if q = s 0 [v i ],
then the proposition has two establishers:

s 0 [v i ] and T = Last(δ v i (s 0 [v i ], s 0 [v i ])). Unlike (2), the proposition q has two clobberers: C 1 and C 2 such that C 1 = F irst(δ v i (s 0 [v i ], s 0 [v i ])) and C 2 = F irst(δ v i (s 0 [v i ], s [v i ])
). The chain between s 0 [v i ] and s [v i ] can be written:

chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], s 0 [v i ]) ⊕ δ v i (s 0 [v i ], s [v i ]) = C 1 , ..., T ⊕ C 2 , ..., a s [v i ] v i So, if q = s 0 [v i ],
then there are two possibilities to assert a.

• The action a is established by the start and promoted before C 1 and before C 2 . Because C 1 is already ordered before C 2 in the chain of actions, the ordering of a with C 1 only is sufficient:

C 1 = F irst(δ v i (s 0 [v i ], s 0 [v i ])), a ≺ C 1 (4.8)
• The action a is established by T, demoted after C 1 and promoted before C 2 . Similar to the equation (4.7), the demotion with C 1 is not necessary because C 1 is already ordered before T via

δ v i (s 0 [v i ], s 0 [v i ]). Eventually: ∧ T = Last(δ v i (s 0 [v i ], s 0 [v i ])), T ≺ a C 2 = F irst(δ v i (s 0 [v i ], s [v i ])), a ≺ C 2 (4.9) If q = s 0 [v i ] and q = s [v i ]
then the assertion of the action a follows the rule given by the equation (4.3). If q = s [v i ], it follows that of the equation (4.5).

It ensues from these different cases the following definition for the Υ 1 -Truth Criterion.

Theorem 4.3.1 (Υ 1 -Truth Criterion). Let ∆ = (A, ≺) be a Υ 1 non-linear solution plan for the SAS-PU problem instance = (Φ, s 0 , s ) and σ v i = (A(v i ), ≺) a chain of actions affecting v i ∈ M. ∆ is a Υ 1 correct non-linear plan iff, For each v i ∈ M, 1. σ v i = chain v i (s 0 [v i ], s [v i ]) ⊂ ∆ ⇒ σ v i is Υ 1 or empty. 2. ∀a ∈ A \ A(v i ), p = prv(a)[v i ], (∃T ∈ σ v i , p = post(T)[v i ] ∧ T ≺ a) ∧(∃C ∈ σ v i , p = pre(C)[v i ] ∧ (T ≺ C ∨ p = s 0 [v i ])) ⇒ (C is unique ∧ a ≺ C)
Proof. A SAS-PU solution plan is a collection of chain of actions with additional relations between the chains via the prevail-conditions. A solution plan is therefore Υ 1 if each non-empty chain of actions composing it is Υ 1 . The second part is an adaption of Theorem 1 [33, p.4]

σ v i . • Equation (4.8): a ≺ C 1 ⇒ a ≺ C 2 , because C 1 is ordered before C 2 in σ v i . • Equation (4.9): T ≺ a ∧ a ≺ C 2 ⇒ C 1 ≺ T, because C 1 is already ordered before T in σ v i .
It results there is a unique clobberer that needs an additional order with the action to assert, and this unique additional order is set by promotion.

This truth criterion seems to be adaptable to Υ k , with for each σ v i ⊂ ∆, σ v i is either empty or at most Υ k , and point 2 remains true. Point 2 may be proven by recursion with:

σ v i = chain v i (s 0 [v i ], s [v i ]) = b, ..., T 1 ⊕ C 1 , ..., T 2 ⊕ ... ⊕ C k-1 , ..., T k ⊕ C k , ..., a s [v i ] v i with b ∈ A(v i ), pre(b)[v i ] = s 0 [v i ] That is, the action to assert, denoted a, has k clobberers, denoted C k . Let l ∈ N such that 1 < l ≤ k, if T l establishes a prevail-condition of a, then ∀n ∈ [[1, l -1]], a is demoted after C n , i.e. C n ≺ T l , and ∀m ∈ [[l, k]], a is promoted before C m , i.e. a ≺ C m .
But similar to the argument used in the proof of the Υ 1 -Truth Criterion, if T l establishes a and T l ≺ C l in σ v i , then it is sufficient to promote a before C l . C l is also the unique clobberer that succeeds T l in σ v i . The other clobberers are already properly ordered with respect to T l and C l in σ v i .

It should be notice that the situations described in cases ( 2) and (3) when q = s 0 [v i ] does not exist with the restriction (S). Indeed, in such situations, there is no need for the minimal chain

δ v i (s 0 [v i ], s 0 [v i ]
) to be built, because (S) insures there is a unique subgoal in v i , which, here, is the start. So,

δ v i (s 0 [v i ], s 0 [v i ]) =
increases the length of the chain of actions for nothing because it passes through no subgoal. It results that A will not be minimal if actions of

δ v i (s 0 [v i ], s 0 [v i ]
) are added, thus contradicting C. Bäckström's IMP. The situation of cases ( 2) and ( 3)

when q = s 0 [v i ] therefore occur iff |R O v i | ≥ 2 and q ∈ R O v i , q = s 0 [v i ] and if δ v i (s 0 [v i ], s 0 [v i ]) passes through other requested values in R O v i \ {q} to solve the problem. Corollary 4.3.1. If q = s 0 [v i ] is a proposition to assert in a Υ 1 solution plan ∆ and it exists a q v i ∈ ∆ s.t. ope(a q v i ) ∈ Cycle(v i )
, then q is either established by the start or by the unique establisher in ∆, namely a q v i . Otherwise, q has a unique establisher:

the start if q = s 0 [v i ] a q v i if q = s 0 [v i ]
Proof. Direct from the explanation of the equations (4.3) to (4.9).

A New

p v i ∈ O, if o p v i ∈ RO ∧ o p v i ∈ Cycle(v i ), then |Cycle(v i )| ≤ k.
C k is a restriction on structures of the class SAS-PU, so it is necessary combined with (P) and (U). The restriction C k is obviously compatible with the restriction (S) as well. For example, the Horse Breeder and the Gray Code problems are of the class SAS-PUC 2 :

• RO HB = {TakeHaystack, DropBucket, TakeBucket} with TakeHaystack ∈ Cycle(v 0 ) and DropBucket, TakeBucket ∈ Cycle(v 1 ). Therefore, only Cycle(v 0 ) and Cycle(v 1 ) are affected by the restriction C

2 : |Cycle(v 0 )| ≤ 2, |Cycle(v 1 )| ≤ 2. Cycle(v 2 )
is not affected by the restriction. Even though it is empty, its size could have been greater than 2.

•

RO GC = {o - v 0 , o + v 0 , o + v 1 } with o - v 0 , o + v 0 ∈ Cycle(v 0 ) and o + v 1 ∈ Cycle(v 1 )
. Therefore, only these two cycles are affected by the restriction C

2 : |Cycle(v 0 )| ≤ 2, |Cycle(v 1 )| ≤ 2.
It should be noticed that SAS-PUC 1 problems do not exist due to the restriction (S4) of the SAS-structure. Let v i ∈ M, if an operator o is requestable and inside Cycle(v i ), C 1 imposes that: Cycle(v i ) ≤ 1, i.e. Cycle(v i ) = {o} which means o is looping with itself, which is not possible due to (S4):

pre(o)[v i ] = u ⇒ pre(o)[v i ] = post(o)[v i ].
In the following, I therefore study SAS-PUC k problems with k = 0, k = 2 and k ≥ 3. I first introduce some useful theorems and lemmata for what follows. In the following, if a problem is exactly of the class SAS-PUC k , then the problem is in SAS-PUC k \ SAS-PUC k-1 . In other words, the structure of such a problem possesses at least one cycle with k operators in it and one of them at least is requestable. Theorem 4.4.1 implies that a problem that is SAS-PUC 2 is also a SAS-PUC 3 problem, i.e. it is a SAS-PUC k problem with k ≥ 2. This precision is important for what follows because, depending on k ∈ N, problems that are exactly of the class SAS-PUC k may have characteristics that problems of the class SAS-PUC k+1 do not have.

1. SAS-PUC (k-1) ⊂ SAS-PUC k 2. SAS-PUSC k ⊂ SAS-PUC k Proof. Let v i ∈ M and k ∈ N, 1. Let Φ = (M, S,

Theorem 4.4.2. The limit of the class SAS-PUC k as k approaches infinity is the class SAS-PU:

lim k→∞ (SAS-PUC k ) = SAS-PU Proof.
According to Lemma 4.2.2, the domain operator graphs of (PU) operators are either a directed tree or a directed graph with a single cycle. The class SAS-PU accepts all these domain operator graphs whereas the restriction C k discards those whose unique cycle has more than k nodes. Thus, the higher k, the fewer cyclic domain operator graphs discarded. Therefore, when k approaches infinity, SAS-PUC k accepts all the domain operator graphs which implies: In the following subsections, I first present the class of problems SAS-PUC 0 and prove that all these problems are Υ 1 . I then present the class of problems SAS-PUC 2 and prove that this class is generally intractable. With additional restrictions, I present SAS-PUC S 2 and SAS-PUC * 2 problems which are subclasses of SAS-PUC 2 and whose problems are Υ 1 . Finally, I prove that problems that are exactly of the class SAS-PUC k with k ≥ 3 are at least Υ 2 . That is, every SAS-PUC k problem with k ≥ 3 has at least one problem instance whose minimal solution plan is Υ 2 .

lim k→∞ (SAS-PUC k ) = SAS-PU.

The Class of Problems SAS-PUC

0 Theorem 4.4.3. SAS-PUC 0 problems are Υ 1 . Proof. Let v i ∈ M, p ∈ R O v i a requestable value. C 0 implies that there is no requestable operator in a cycle: ∀o p v i ∈ RO, ∀v i ∈ M, o p v i / ∈ Cycle(v i ).
Therefore, every chain of actions in a minimal solution plan solving an instance = (Φ, s 0 , s ) is of the form:

σ v i = chain v i (s 0 [v i ], s [v i ]) = δ v i (s 0 [v i ], p) ⊕ δ v i (p, s [v i ]) = δ v i (s 0 [v i ], s [v i ])
If there is a requestable value p, then the establisher of p is either the start or an action of σ v i . Otherwise, it means the establisher must be instantiated from a requestable operator that are in the cycle of the domain operator graph, which cannot occur due to C 0 . Therefore, for each v i ∈ M, if σ v i is a non-empty chain of actions of the plan, then it is the minimal chain from the start to the goal:

δ v i (s 0 [v i ], s [v i ])
. Therefore, every non-empty chain of a plan solving an instance is Υ 1 . It implies that the minimal solution plan of every solvable instance of any SAS-PUC 0 problem is Υ 1 .

This class of problems does not prevent the use of the unique cycle of the domain operator graph, but prevents the definition of requestable operators inside that cycle. It results that actions establishing prevail-conditions, which are instances of requestable operators, will always be part of the minimal action chain between the start and the goal state, which is unique (cf. Lemma 3.2.2) and Υ 1 (cf. Lemma 3.2.1).

It also results that every problem structure whose domain operator graphs are directed trees (cf. Lemma 4.2.2), i.e. ∀v i ∈ M, Cycle(v i ) = ∅, is of the class SAS-PUC 0 . Proof. The restriction C 2 restricts the number of operators to at most 2 in the unique cycle of domain operator graphs that possess requestable operators. The restriction (B) restricts every domain operator graph to only have two operators, i.e. two nodes. In other words, (B) implies that, for each

The Class of Problems SAS-PUC

v i ∈ M, |O(v i )| ≤ 2 while the restriction C 2 implies |Cycle(v i )| ≤ 2. As Cycle(v i ) ⊆ O(v i ),
o 1 v 0 v 0 = 0 v 0 = 1 u, 1, u o 0 v 1 v 1 = 1 v 1 = 0 u, u, u o 1 v 1 v 1 = 0 v 1 = 1 u, u, u o 1 v 2 v 2 = 0 v 2 = 1 u, 0, u Table 4.4: Operators of a SAS-PUC 2 structure Φ. = (Φ, s 0 = 0, 0, 0 ,s = 1, 1, 1 ) is a possible problem instance.
There are exactly 64 different problem instances with this structure (2 3 different starts times 2 3 different goals), 40 of which are solvable. It can be proved by hand or with Algorithm 4 that each of the solvable problem instances has a minimal solution plan that is Υ 1 . Therefore, it exists at least one problem of the class SAS-PUC 2 that is Υ 1 . In other words, it may exist subclasses of SAS-PUC 2 whose problems are Υ 1 . The following two subsections present such subclasses. • it is restricted by C 2 ;

The Class of Problems SAS-PUC S

• there is at most one requestable operator per domain operator graph cycle.

Let v i ∈ M such that Cycle(v i ) = ∅ is a cycle affected by the structural restriction C 2 , the size of Cycle(v i ) is therefore: |Cycle(v i )| = 2. Let denote o - v i and o + v i the two operators of Cycle(v i ) such that Cycle(v i ) = {o - v i , o + v i }.
At least one of the two, let consider o + v i , is a requestable operator (Definition 4.4.1). If Cycle(v i ) respects the structural restriction C S 2 , then o + v i is the single requestable operator of Cycle(v i ). The notation C S 2 is directly inspired from the restriction (S). The main difference with the syntactical restriction (S) is that the set of requestable value for a given state variable can be greater than 1. Let v i ∈ M, we can have:

R O v i = {+, p 1 , ..., p n}, with n ≤ |D v i | -2
such that:

o + v i ∈ RO(v i ) ∪ Cycle(v i ) ∧ ∀k ∈ [[1, n]], o p k v i ∈ RO(v i ) \ Cycle(v i )
That is, each requestable operator o p k v i is not in the cycle of the domain operator graph, which implies these operators can only have at most one action instance each, no matter the problem instance. It remains to show that o + v i , the single requestable operator of Cycle(v i ), can also be instantiated at most once, no matter the SAS-PUC S 2 problem instance.

In SAS + -PUS problems, |R O v i | ≤ 1 for each v i ∈ M. Let denote p the single value of the domain D v i , such that R O v i = {p}.
When defining the IMP, C. Bäckström described p as a subgoal that each planner respecting the IMP may have to consider in order to have a correct minimal solution plan for a given SAS + -PUS problem instance. With D. Chapman's terms, p is a proposition to assert, i.e. p must be established and declobbered if it is a prevail-condition of an action a in the solution plan. If I apply these terms to each p ∈ {+, p 1 , ..., p n}, then there are all possible propositions to assert before reaching the goal state.

Definition 4.4.3. v i ∈ M, x, y ∈ R O v i s.t. x = y. δ v i (x, y) is a meaningful minimal chain if x is a clobbering situation and y is an establishing one. Let R O v i = {+, p 1 , ..., p n} such that RO(v i )∪Cycle(v i ) = {o + v i } and RO(v i )\Cycle(v i ) = {p 1 , ..., p n}. For 1 ≤ k < n, let consider o p k v i ≺ o p k+1 v i in G O (v i ). It implies that o + v i ≺ o p 1 v i in G O (v i ). It results from Definition 4.4.3 that: • δ v i (s 0 [v i ], s [v i ]) is meaningful. • ∀p ∈ RO(v i ), δ v i (s 0 [v i ], p) and δ v i (p, s [v i ]) are both meaningful. • ∀p i , p j ∈ {p 1 , ..., p n}, δ v i (p i , p j ) is meaningful iff 1 ≤ i < j ≤ n. And δ v i (p j , p i ) = .
• ∀p ∈ {p 1 , ..., p n}, δ v i (+, p) is meaningful and δ v i (p, +) = .

I used this notion of meaningfulness to prove that SAS-PUC S 2 problems are Υ 1 .

Theorem 4.4.5. SAS-PUC S 2 problems are Υ 1 .

The proof is provided in Appendix C.1.

In comparison with the restriction (S), the structural restriction C S 2 , although less restrictive for operators outside the cycle of their domain operator graph, does not allow On/Off situations neither. Problems of the class SAS-PUC S 2 do not allow two requestable operators affecting the same state variable to be in the unique cycle of their domain operator graph. So in a problem instance, if an action required the switch to be on and another action the switch to be off, then there is one operator to instantiate "turn on" and one operator to instantiate "turn off". The two operators create a cycle and are both requestable, therefore the problem is not C S 2 . The main purpose of this class of problems, however, is to prove the following class.

The Class of Problems SAS-PUC *

2

A consequence of the Υ 1 -Truth Criterion (Theorem 4.3.1), is that there is at most one clobberer per prevail-condition of an action that needs an additional order, by promotion, to ensure a Υ 1 correct solution plan. Similar to the predecessor sets N prv (o p v i ) and N pre (o p v i ) of an operator o p v i , we can thus represent a set of predecessors for o p v i that represents the set of operators whose instance in a Υ 1 solution plan may need to be ordered after an instance of another operator o q v j . That is, an instance of o q v j may be promoted before an instance of

o p v i . Definition 4.4.4 (N clob (o p v i )). Let o p v i ∈ O, N clob (o p v i )

is the set of predecessors whose instance may be clobbered by an instance of o

p v i : N clob (o p v i ) = {o q v j | v i = v j ∧ o q v j / ∈ N prv (o p v i ) ∧ pre(o p v i ) = prv(o q v j )[v i ]}.
(O) = {(o q v i , o p v i ) | o q v i ∈ N clob (o p v i )}
, the set of directed edges representing the order by promotion between the operator to assert o q v i and its clobberer o p v i . The set of directed edges E clob (O) will be colored in red in Figures representing declobbered operator graphs.

• if it exists o x v i , o y v i ∈ RO(v i ) s.t. o x v i , o y v i ∈ Cycle(v i ) and for all o 1 , o 2 ∈ O \ O(v i ) s.t. prv(o 1 )[v i ] = x ∧ prv(o 2 )[v i ] = y then: o 1 is not related to o 2 in G O\Cycle(v i )
Let illustrate C * 2 with the Horse Breeder (HB) and the Gray Code (GC), the nodes of the two graphs in Figure 4.5 have three colors: yellow, green and blue. The yellow color is for cycle with at least two requestable operators, i.e. it colors o x v i and o y v i from the definition:

• Yellow HB = {PickupBucket, DropBucket}

• Yellow GC = {o + v 0 , o - v 0 } Due to C 2 ,
the yellow cycle has exactly two operators in it and it is two requestable operators. They therefore precede other operators in O \ O(v i ) by post-prv dependency, and these operators are colored either in blue or in green depending on the requestable operator that precedes them. We have:

• Blue HB = {TakeHaystack}, it is preceded by DropBucket. • Green HB = {FillBucketWithWater, FillHorseTrough}, they are preceded by TakeBucket.

• Blue GC = {o - v 2 , o + v 2 }, they are preceded by o - v 0 . • Green GC = {o - v 1 , o + v 1 }, they are preceded by o + v 0 .
These blue and green operators represent o 1 and o 2 respectively in the definition of C * 2 . In order to see if the Horse Breeder and the Gray Code structures respect the restriction C * 2 , the domain operator graph that contains the yellow operators is removed from the declobbered operator graph and we then check the relation between the green operators and the blue ones. Figure 4.6 shows the declobbered operator graphs with the domain operator graphs containing the yellow nodes removed of both the Horse Breeder and the Gray Code. The resulting graphs are respectively denoted G HB and G GC .

• The Horse Breeder structure respect C * 2 : each blue operator is not related to any of the green operator in G + O HB \ Cycle(v 1 ).

• The Gray Code structure does not respect C * 2 : each blue operator is preceded by the green operators and each green operator is preceded by the blue operators in

G + O GC \ Cycle(v 0 ). For instance, o + v 2 ≺ o - v 1 and it exists the sequence of operators o - v 1 , o + v 1 , o + v 2 in G + O GC \ Cycle(v 0 ), which implies o - v 1 ≺ o + v 2 by transitivity. Hence C * 2 is not respected.
The idea of this incremented restriction C * 2 is to keep the relaxation of (S) and to prevent the creation of problems whose instances may require the planner to loop through the unique cycle of some domain operator graphs. If two operators o 1 and o 2 with a different defined prevail-condition on a state variable (so the problem is not (S)) are related to each other in some ways, and if the requestable operators that solve these prevail-conditions are in the unique cycle of their domain operator graph, then is exists problem instances where one of the requestable operators needs to be instantiated at least twice in order to respect the existing order between o 1 and o 2 in the declobbered operator graph. In appendix C, I provide four operator graphs that could represent four possible SAS-PU structures to illustrate the issue when o 1 and o 2 are related. In fact, we saw in subsection 4.3.2 that a Υ 1 chain of actions implies that, for each v i ∈ M, a situation p ∈ D v i is most of the time established once (cases 1, 2 and 3) and is established twice iff p = s 0 [v i ] (cases 2 and 3). As the start s 0 [v i ] is not an action, in every case there is at most one action in a Υ 1 chain that can establish p. As a consequence, every action instance of operators that has p as a prevail-condition must be ordered after the establisher of p (For examples, see the blue actions in the action graphs of Figures C.2c Proof. The idea of the proof is to show that each problem that are exactly of the class SAS-PUC k , with k ≥ 3, has a problem instance whose minimal solution plan is Υ 2 . To do so, I define a scalable problem instance in function of k whose minimal solution plan is Υ 2 , thus proving that SAS-PUC k problems, with k ≥ 3, are at least Υ 2 .

The scalable problem instance in function of k, with k ≥ 3, is defined as follows. Each problem that are exactly of the class SAS-PUC k has at least one domain operator graph with a cycle of the size k, and in this cycle, at least one operator is requestable. Let consider Cycle(v i ) as the one respecting:

|Cycle(v i )| = k. Cycle(v i ) = {o 0 v i , .., o k-1 v i } = {o l v i | pre(o l v i ) = l -1 if l > 0, pre(o 0 v i ) = k -1, otherwise.}. As k ≥ 3, then {o 0 v i , o 1 v i , o 2 v i } ⊆ Cycle(v i ) and at least one of them is requestable. If one of them is requestable, let say o 0 v i , then it exists an operator o ∈ O \ O(v i ) such that prv(o )[v i ] = post(o 0 v i ).
Let consider v j ∈ M being the state variable affected by o'. The scalable problem instance is: This problem instance is solved by ∆ such that, a is an action instance of o :

k = (Φ, s 0 , s ) with:      s 0 [v i ] = 1, s [v i ] = 2 s 0 [v j ] = pre(o ), s [v j ] = post(o ) s 0 [v l ] = s [v l ], ∀v l ∈ M \ {v i , v j }
∆ = δ v i (s 0 [v i ], prv(o )[v i ]) ⊕ a ⊕ δ v i (prv(o )[v i ], s [v i ]) = δ v i (1, 0) ⊕ a ⊕ δ v i (0, 2) = δ v i (1, 2) ⊕ δ v i (2, 0) ⊕ a ⊕ δ v i (0, 1) ⊕ δ v i (1, 2)
And ∆ is Υ 2 because δ v i (1, 2) appears twice. Let a p v i the action instance of o p v i , this problem instance is scalable in function of k because |δ v i (2, 0)| = k -2 such that:

• δ v i (0, 1) = a 1 v i • δ v i (1, 2) = a 2 v i • δ v i (2, 0) = a 0 v i , if k = 3 a 3 v i , ..., a k-1 v i , a 0 v i , if k > 3
This problem instance can be created in every problem of the class SAS-PUC k , with k ≥ 3. It concludes the proof.

Because I am interested in problem instance whose minimal solution plan is Υ 1 , it results that I cannot consider SAS-PUC k problems with k ≥ 3.

TopoPlan P

This section presents the algorithm TopoPlan, denoted P, which solves SAS-PUC k problems that are at most Υ 1 . Each problem instance are denoted = (Φ, s 0 , s ). P is based on P: it plans backwards to take advantage of the post-unicity and does it in three phases.

Phase 1 builds the minimal chains of actions between the start s 0 and the goal s for each state variable v i ∈ M. All the actions of these chains are coloured yellow ; Phase 2 checks the prevail-conditions of all the yellow actions. It creates orders between the yellow actions of the different chains so as to establish and/or promote each defined prevail-condition of each the yellow actions correctly. Let v i ∈ M and p ∈ R O v i a prevail-requestable value, two minimal chains of actions, one between s 0 [v i ] and p, and one between p and s 0 [v i ] may be built to ensure the establishment of p while respecting the Υ 1 -Truth Criterion (cf. Cases (2) and ( 3) in subsection 4.3.2). Eventually, a minimal and Υ 1 non-linear plan is returned ; Phase 3 topologically sorts the non-linear plan. P differs from P in several ways, however: I added extra conditions to take into account the relaxation of the restriction (S) ; I went deeper in the pre-processing and created additional sets of operators (cf. Definition 4.1.3) to reduce the complexity of P ; The pre-processing of P is not done while planning compared to P, but while compiling ; Finally, P does not instantiate actions, it colours the operators and returns plans of identifiers.

The section is organised as follows: I first describe the pre-processing, then I describe the algorithm and give complexity theorems. It follows a step by step explanation of the resolution of two problem instances of the Horse Breeder by P. First of all, I described operators in an external file, namely a .txt file, which is also a technique used by developers. In F.E.A.R. for instance, new assets, operators included, are described in a dedicated application called "GDBEdit" (which stands for Game DataBase Editor) that creates .record files used by the compiler [10]. Figure 4.8 shows an example of my .txt files, it is the Horse Breeder from the RDR-2 benchmark (described in subsection 5.2.1). The first column, coloured in gray, numbered the lines. The first line stores the number of state variables (l.1, m = 3), then there is the number of operators (l.2, |O| = 7), followed by the maximum number of values in a state variable domain (l.3, n = 3). Each line, from l.4 to l.10, is the representation of a (PU) operator: the first column (without considering the gray one) is the index of the affected state variable ; the second column stores the value of the pre-condition ; the third column is the post-condition ; the 4 th column to the (4 + m) th column are the prevail-conditions. In the Horse Breeder example, there are 3 state variables so there are 3 prevail-conditions per operator. u is the undefined value ; The last column stores the name of each operator. Is called Setup the pre-processing phase of my algorithm, which is done prior to any planning sessions. That is, the setup is performed while compiling the code. The .txt file is translated to create a SAS structure Φ = (M, S, O). Each operator gets the state variable it affects, the value of its pre-and post-condition, and finally gets its prevail-conditions. They also get an identifier (cf. Definition 4.1.1), and their predecessor sets N pre and N prv are built (cf. Definition 4.1.3). Operators are stored in a hashing table, in the position given by their identifier. All these steps can be done line per line while the setup is browsing the .txt files.

Setting up the Problem

Once each operator has been configured, then the structure is verified to see whether

the restriction C 0 , C S 2 or C * 2 is respected. It is done in 4 steps: 1. Mark every operator o p v i ∈ O such that o p v i ∈ Cycle(v i ) and o p v i ∈ RO. (a) If o p v i is marked and |Cycle(v i )| > 2, then the problem is C k with k > 2.
The structure is not valid. (cf. Lemma 4.4.1).

2. If no operator has been marked, then the structure is C 0 and therefore valid (cf.

Theorem 4.4.3).

Else if exactly one operator per cycle has been marked

, then the structure is C S 2 , i.e. it is valid (cf. Theorem 4.4.5).

Else Let Cycle(v

i ) = {o - v i , o + v i } and o - v i , o + v i ∈ RO(v i ), (a) If for each pair o, o ∈ O such that prv(o)[v i ] = + and prv(o )[v i ] = -, o is related to o in G O\Cycle(v i )
, then the problem does not respect C * 2 . (b) Else, the problem is C * 2 and therefore valid (cf. Theorem 4.4.6). At last, I equipped actions with a getter called Next() whose role is to return in constant time their successor of in the chain. First of all, because of Υ 1 , each operator is instantiated at most once in the solution plan, it results that actions, which are instances of an operator, can be identified with the same identifier as their operator (cf. Definition 4.1.1). It results that establishers, which are characterized by the post-condition (cf. Definition 3.2.5), are easily identified and can therefore be addressed in constant time while planning. Clobberers, on the contrary, are characterized by their pre-condition and postuniqueness does not imply pre-uniqueness. This can be highlight with the Horse Breeder example (Table 3.1), DropHaystack and FillFeeder are post-unique but they share the same pre-condition. It results that clobberers cannot be identified the same way as establishers. Let q ∈ R O v i be a requestable value on v i ∈ M such that T and C respectively are the establisher and the clobberer of q, it can be highlighted that:

post(T)[v i ] = pre(C)[v i ] = q. Due to Υ 1 , if q = s 0 [v i ], σ v i = chain v i (s 0 [v i ], s [v i ]
) passes through q exactly once. As explained in subsection 4.3.2, q is established by T and promoted before C, the equality

post(T)[v i ] = pre(C)[v i ]
therefore implies that C succeeds T in σ v i . As T can be identified in O(1), I therefore introduce Next(), a getter that is dynamically set, to return in constant time the successor of T in σ v i . Next() must be dynamically set because an operator may have several successor via the post-pre dependencies but an action only has one in the chain. So, for each action in A, Next() is set while building the chain of actions (cf. procedure 2 BuildChain).

Algorithm

In this subsection, I presents my algorithm TopoPlan, denoted P. Due to the output restriction Υ 1 , the size of the action set A in a minimal solution plan is at most equal to the size of the operator set O: |A| ≤ |O|. In addition, because each operator of O has at most one action instance in A, the action instance and its operator can both share the same identifier while planning (cf. Definition 4.1.1). Let v i ∈ M, p ∈ D v i , I therefore use the identifier a p v i to refer to the operator o p v i ∈ O and its action instance

a p v i ∈ A (so, ope(a p v i ) = o p v i ).
Similarly, hoping that the reader will not be confused, I will only use the term action and the notation a p v i in the following. For example: a p v i ∈ O and a p v i ∈ A. To avoid having to browse A to know if an action is inside, I implemented a labeling technique with colors such that:

• (White) the action is not instantiated, i.e. it is not in A. Every action of O is white at the beginning of the resolution of any problem instance ;

• (Yellow) the action is instantiated, i.e. it is in A ;

• (Blue) the action is being topologically sorted ;

• (Green) the action is topologically sorted, i.e. added in a linear solution plan ∆.

The identifier a p v i refers to an action of the hashing table that was built in the setup. The use of colors with the hashing table therefore allows to know in constant time the state of an action. P plans backwards using the identification technique a p v i ∈ O and the predecessor sets N pre (a p v i ) and N prv (a p v i ) to find the minimal solution plan ∆ that solves an instance = (Φ, s 0 , s ) of the class SAS-PUC 0 , SAS-PUC S 2 or SAS-PUC * 2 (cf. Definitions 4.1.1 and 4.1.3). The resolution of by P consists in building A and a sufficient set of orders between the actions of A, denoted E A , while respecting the Υ 1 -Truth Criterion. P relies on the procedure 2 (BuildChain) to build A. Definition 4.5.1. The specification of BuildChain is:

• Input: O, A, E A , v i ∈ M, s, g ∈ D v i s.t. s = g and a g v i ∈ O is white. • Output: δ v i (s, g). Each action a of δ v i (s, g) is colored yellow, added to A and each order (b, a), s.t. b ∈ N pre (a), is added to E A .
is a non-linear sequence of actions and its output specification is the linearisation of that sequence of actions, if it exists. In the algorithm TopoPlan, if the problem instance is solvable, the input of DFSTopo is a Υ 1 , non-linear and minimal solution plan that respect the Υ 1 -Truth Criterion. Therefore, DFSTopo returns a Υ 1 , linear and minimal solution plan and it still respects the Υ 1 -Truth Criterion. If the problem instance is not solvable, Phase 2 may return a non-linear sequence of action that is not irreflexive. DFSTopo therefore yields a failure thanks to the blue color that spots looping issue.

Definition 4.5.2. The specification of TopoPlan is:

• Input: = (M, O, s 0 , s ), a problem instance of the class SAS-PUC 0 , SAS-PUC S 2 or SAS-PUC * 2 .
• Output: If is solvable, then P returns ∆ a linear, Υ 1 and minimal solution plan that solves while respecting the Υ 1 -Truth Criterion. If is not solvable, P yields a failure. Theorem 4.5.2. The algorithm TopoPlan satisfies its specification, so it is correct and complete.

Proof. The proof is given in Appendix D.

D v i . 1: x ← ∅; y ← ∅ 2: if a g v i is a ghost action then fail 3: end if 4: Color(a g v i ) ← yellow; A ← A ∪ {a g v i }; y ← pre(a g v i ); E A ← E A ∪ {(a y v i , a g v i )} 5: if Next(a y v i ) = ∅ {a y v i can
Color(a y v i ) ← yellow; A ← A ∪ {a y v i }; x ← y; y ← pre(a y v i ); E A ← E A ∪ {(a y v i , a x v i )} 14:
if Next(a y v i ) = ∅ {a y v i can be a ghost action.} then 15:

Next(a y v i ) ← a x v i 16:
end if 17: end while Procedure 3 DFSTopo(a p v i , A, E A , s 0 , ∆) Input: a p v i , a yellow action affecting v i with the value p ∈ D v i ; s 0 ∈ S; ∆, the totally ordered solution plan. Output: a p v i is colored in green once all its neighbors have been topologically ordered; It is then enqueued to ∆.

1: Color(a p v i ) ← blue 2: for a q v j ∈ E A (a p v i ) do 3: if a q v j / ∈ N pre (a p v i ) ∨ a p v i is not the first action to modify s 0 [v i ] then 4:
if Color(a q v j ) = blue then fail {Cycle spotted.}

5:

end if

6:
if Color(a q v j ) = yellow then 7:

DFSTopo(a q v j , A, E A , s 0 , ∆) Parameters: A, the set of yellow actions ; E A , the set of orders between yellow actions.

1: ∆ ← ∅; A ← ∅; E A ← ∅ 2: for v i ∈ M do {Phase 1} 3: if s 0 [v i ] = s [v i ] then 4: BuildChain(v i , s 0 [v i ], s [v i ], A, E A , O)
for a q v j ∈ N prv (a p v i ) do 11: if q = s 0 [v j ] then 12: if a q v j / ∈ O then fail 13:
end if

14:
if Color(a q v j ) = white then 15:

BuildChain(v j , s 0 [v j ], q, A, E A , O) 16:
end if

17:

E A ← E A ∪ {(a q v j , a p v i )} 18:
end if 19:

if q = s [v j ] then 20:
if Next(a q v j ) = ∅ then 21:

BuildChain(v j , q, s 0 [v j ], A, E A , O) 22:
end if

23: if prv(Next(a q v j ))[v i ] = p then 24: E A ← E A ∪ {(a p v i ,Next(a q v j ))} 25:
end if

26:

end if 27:

if q = s 0 [v j ] ∧ a q v j ∈ Cycle(v j ) ∧ Cycle(v j ) is concerned by C * 2 then
28:

E A ← E A ∪ {(a s 0 [v j ] v j , a p v i )} 29: end if 30:
end for 31: end for 32: for a ∈ A do {Phase 3} Two examples on how P resolves a problem instance are given in Appendix E.

Discussion

• TopoPlan (P) is based on C. Bäckström's algorithm that solve SAS + -PUS problems. I have not considered the SAS + -structure, however. The next step would be to consider the SAS * -structure, i.e. to allow undefined state variables in the goal state. I believe P can be used as a good basis to consider solving SAS * -PUC 0 , SAS * -PUC S 2 or SAS * -PUC * 2 problem instances. It may be useful to add a new color than yellow to mark actions inside cycles, though, so that yellow can be used by BuildChain as a stop condition to concatenate chains of actions instead of yielding a failure. And the other color, on the contrary, can be used as a stop condition to yield a failure as we do not want BuildChain to loop inside the unique cycle of each domain operator graph. Indeed, an undefined state variable in the goal state does not mean the value given by the start cannot be changed. But not all values are meaningful, the meaningful one are the requestable values. So, to consider the SAS * structure, it is necessary to know the last requestable value required by the solution plan solving a given problem instance. The action chain from the start to the last useful requestable value can thus be built by BuildChain incrementally, hence the use of two colors. If the incremental construction takes place inside the cycle of the domain operator graph, then it probably means the planner is looping, so a stop condition is required. If the incremental construction takes place outside the cycle, then the planner must ensure it is not building several different branches of the domain operator graph, but only one.

• If the SAS + -structure is considered with the restriction (P), (U) and (C k ), I believe further restrictions must established, in particular to avoid the construction of several operators with an undefined precondition and affecting the same state variable. That is,

O = {o k v i | v i ∈ M, k ∈ D v i , pre(o k v i ) = u}.
This kind of operator sets are useful to instantiate movement actions for instance: o k v i = MoveTo(k), such that pre(MoveTo(k)) = u. But movement operators/actions are related to pathfinding, which is known to be hard.

• TopoPlan can be used to solve SAS-PUS problem instance, but only those whose minimal solution plan is Υ 1 . I haven't proved theoretically this assertion, however. But any solution plan returned by P can be verified with P.

• Concerning (3.27) to (3.29) in P, it only concerns operators of cycles strictly restricted by C * 2 because operators of C S 2 or C 0 are never concerned by the double establishment situation described in Case 1 and Case 2 (subsection 4.3.2) when the proposition to assert is equal to the start. v i ∈ M, action chains in minimal solution plan solving SAS-PUC 0 problem instance are of the form of

σ 1 = δ v i (s 0 [v i ], s [v i ]
). Therefore, if the proposition to assert q is equal to the start (q = s 0 [v i ]), then q only needs to be promoted before F irst(σ 1 ). This is handled by P between (3.19) and (3.26). For SAS-PUC S 2 problem instances, minimal solution plans resemble those of SAS-PUS problem instances that are Υ 1 . The action chain can be of the form of

σ 1 , σ 2 = δ v i (s 0 [v i ], s 0 [v i ]) or σ 3 = σ 2 ⊕ σ 1 . But if σ 2 is built in SAS-PUC S
2 or SAS-PUS problem instances, this is to reach q inside the cycle, so that:

σ 2 = δ v i (s 0 [v i ], q) ⊕ δ v i (q, s 0 [v 1 ])
Otherwise, if q = s 0 [v i ] or outside the cycle, then σ 2 is never built because it is meaningless and therefore breaks the minimality of the solution plan if it is built. It results that if q = s 0 [v i ], then q is always promoted before σ 1 .

In addition to that, the if condition (3.27) and the promotion order (3.28) is only useful for cycles that have two requestable operators, hence cycles that are C * 2 . Let v j ∈ M, Case 2 and Case 3 (subsection 4.3.2) mention that a proposition q, such that q = s 0 [v j ], can be established in two ways: by the start itself, or by the action a q v j , i.e. the establisher of q. (3.28) is therefore a choice as I could have considered:

E A ← E A ∪ {(a p v i , F irst(δ v i (s 0 [v j ], s 0 [v j ]))}, if F irst(δ v i (s 0 [v j ], s 0 [v j ]) is not an establisher of a p v i .
(4.10) instead of (3.28):

E A ← E A ∪ {(a q v j , a p v i )} (4.11)
The instructions between (3.27) to (3.29) work for SAS-PUC * 2 problem instances because of the structural properties induced by the restriction C * 2 . It may exists SAS-PU problem instances whose minimal solution plan is Υ 1 and where Case 2 and Case 3 with q = s 0 [v j ] may be handled differently. That is, it may exist problem instances where the order (4.10) must be applied over (4.11), and vice versa. But this is not the case with C * 2 .

• P works with SAS-PUC 0 , SAS-PUC S 2 and SAS-PUC * 2 problem instances, which means that it does not exist domain operator graph cycles with more than 3 operators and at least one of them requestable. It results the BuildChain procedures (3.15) and (3.21) will always return a minimal action chain of size 1. These procedures aim at building:

σ 2 = δ v i (s 0 [v i ], s 0 [v i ]) = δ v i (s 0 [v i ], q) (3.15) ⊕ δ v i (q, s 0 [v i ]) (3.21)
Such that q is a requestable value established by a q

v i . Therefore, |Cycle(v i )| = |σ 2 | = 2 and |δ v i (s 0 [v i ], q)| = |δ v i (q, s 0 [v i ])| = 1,
hence the assertion that these BuildChain procedures will always return a chain with one action.

• Eventually, TopoPlan has a linear-time complexity thanks to the predecessor sets (N pre and N prv ) that were created during the setup. I will present experiments in the next chapter and will show that, in addition to reducing complexity, these sets of predecessors prevent P from doing the unnecessary work that P does by checking all the prevail-conditions. In other words, with the predecessor sets P only checks the defined prevail-conditions while P also checks the undefined ones.

Benchmarks and Results

In this chapter, I present the runtime performance of both P and P implemented in C++14 with default settings for Microsoft Visual Studio 2019. All experiments were performed with an AMD Ryzen™ 7 2700X (cf. Table 5.11 for the characteristics), 32Gb of RAM and Windows 10 (64 bits). Unlike C. Bäckström who implemented P with SUN Common Lisp with "no attempts at optimizing it", I personally optimized each of my code: the C++ implementation of P works with vectors and functions' parameters are integers, references and/or pointers in order to avoid copy constructors. Actions are only instantiated once by the FindPost function (cf. Appendix A), and the other functions work with references to address the instantiated actions. Concerning the C++ implementation of P, I remove the vectors in favor of fixed sized arrays. There were some weird results linked to the vectors' dynamic allocation that occurred automatically while the code is being executed. There are therefore a memory management with P, which allowed me to get smooth results. Finally, P also works with pointers and references to keep planning as light as possible.

I present several different benchmarks in this chapter. Let m be the number of state variables and n the size of each state variable domain, the first round of benchmarks are scalable, i.e. they can be scaled in function of m, or in function of n. Their purpose is to show empirically the time complexity of both algorithm P and P. As a reminder, the complexity of

P is O(n • m 2 ) and that of P is O(|A| + |E A |) ≈ O(n • m 2 )
, so both algorithms are affected by the variables m and n. The second round of benchmarks are more realistic, they are based on existing commercial video games. I design them in order to see how many Non-Playable Characters of these benchmarks can get a plan from the C++ implementation of both P and P in real-time.

The chapter is organized as follows: I first introduce the scalable benchmarks and I use them to show empirically that P is indeed quadratic in m and linear in n, and that P is indeed linear in the number of actions plus the number of orders. In a second section, I present three realistic benchmarks based on the existing commercial video games Red Dead Redemption, Assassin's Creed: Origins and Horizon Zero Dawn. I created these benchmarks for two reasons: to test both P and P on more realistic problems and to highlight that it is possible to represent these problems with the new class SAS-PUC * 2 . I end this chapter with a discussion and conclusions on the results of these experiments.

Scalable Benchmarks

In this section, I empirically show the time complexity of both P and P, firstly with respect to m, the number of state variables, and secondly with respect to n, the size of the state variable domains. To do so, I use two scalable instances of 2 new benchmarks designed to have a non-linear minimal solution plan whose size increases in function of the number of state variables (m). The difference between the two problem instances lies in the number of orders between the actions: in the first problem instance, the increase of the number of orders is quadratic with respect to m, while in the second the increase is linear with respect to m. • pre(o

Description of the Scalable Benchmarks

p v i ) = p -1 • post(o p v i ) = p • prv(o p v i )[v j ] = u, for i ≤ j < m • prv(o p v i )[v j ] = n/2 , for 0 ≤ j < i.
MultiPrv_n is of the class SAS-PUSC 0 and, therefore, it can be solved by both P and P.

It is a very constrained problem as there are O(n × m 2 ) orders between the operators. For each v i ∈ M, if all operators of O(v i ) are instantiated to create a chain of actions, we have:

chain v i (0, n -1) = (A(v i ), ≺ pre ) = a 1 v i , a 2 v i , ..., a n-1 v i
The structure of MultiPrv_n_Cycle is equivalent to that of MultiPrv_n, except that, for each v i ∈ M, the operator o 0 v i such that post(o

0 v i ) = 0, pre(o 0 v i ) = n -1 and prv(o 0 v i ) = prv(o p v i ) is added to O(v i ).
It adds the order: a n-1 v i ≺ pre a 0 v i which implies that:

chain v i (0, n -1) ⊕ a 0 v i = chain v i (0, 0)
That is the additional operator and order create a cycle such that O(v i ) = Cycle(v i ).

MultiPrv_2_Cycle is nothing but the Tunnel example of [6, p.17] and MultiPrv_n_Cycle is the generalisation of it. MultiPrv_n_Cycle is of the class SAS-PUSC n . For all n ∈ N, it implies only P is complete on MultiPrv_n_Cycle. P is complete on Tunnel, i.e. MultiPrv_2_Cycle, only. More precisely, if n > 2, for every problem (s 0 , s ) where it exists v i ∈ M such that the single prevail-condition on v i is not in between s 0 [v i ] and s [v i ], then the minimal solution plan is Υ 2 . P will thus return the minimal solution plan whereas P will yield a failure. Otherwise, if the prevail-condition is in between s 0 [v i ] and s [v i ] or equal to one of the two, for each v i ∈ M, the minimal solution plan is Υ 1 and both algorithms return it (cf. Chapter 4). I now describe a possible scalable instance for each of the two problems: MultiPrv_n and MultiPrv_n_Cycle. These scalable instances are the one I used to empirically strengthen my theoretical results on P. For MultiPrv_n, I define the scalable problem instance multi = (Φ multi , s 0 , s ) with:

s 0 = v i = 0 | v i ∈ M ; s = v i = n -1 | v i ∈ M
And for MultiPrv_n_Cycle, mCy = (Φ mCy , s 0 , s ) with:

s 0 = v i = 0 | v i ∈ M ; s = v m-1 = n -1 ⊕ v i = 0 | v i ∈ M \ {v m-1 }
Let consider the scalable problem mCy = Tun3 = (Φ Tun3 , s 0 = 0, 0, 0 , s = 0, 0, 1 ) of MultiPrv_2_Cycle with m = 3, which is the Tunnel planning problem with 3 sections (Φ Tun3 , cf. Figure 5.1). Tun3 states that every section is off initially and the goal is to turn on the third section only. The operator set is described in Table 5.1. The minimal solution

O Tun3 pre() post() prv() Information o 0 v 0 v 0 = 1 v 0 = 0 u, u, u Switch off Sec 1 o 1 v 0 v 0 = 0 v 0 = 1 u, u, u Switch on Sec 1 o 0 v 1 v 1 = 1 v 1 = 0 1, u, u Switch off Sec 2 o 1 v 1 v 1 = 0 v 1 = 1 1, u, u Switch on Sec 2 o 0 v 2 v 2 = 1 v 2 = 0 1, 1, u Switch off Sec 3 o 1 v 2 v 2 = 0 v 2 = 1 1, 1, u Switch on Sec 3 Table 5.1: Operator set of Φ Tun3 . plan of Tun3 is: ∆ = a 1 v 0 , a 1 v 1 , a 1 v 2 , a 0 v 1 , a 0 v 0 , i.e.
to turn on the 3 rd section, the sections 1 to 3 are turned on successively, then the sections 2 and 1 are turned off successively. Then, |∆| = |O| -1, i.e. the planner has to instantiate all the operator once (except one) to solve the problem instance. Both planners, P and P, create 10 orders to solve the problem instance.

It can be proved that both P and P have to create at least (2 • m 2 -3 • m + 1) orders to solve mCy of Tunnel with m sections, and they both return a minimal action plan ∆ of (2 • m -1) actions and that is Υ 1 .

OnePrv_n and OnePrv_n_Cycle

The structures of OnePrv_n and OnePrv_n_Cycle are very close to that of MultiPrv_n and MultiPrv_n_Cycle. They are less constrained, however, as each operator only has one defined prevail-condition. There are O(n×m) orders between the operators. The operators of OnePrv_n follows the following rules, ∀v i ∈ M and 0 < p ≤ n -1:

• pre(a p v i ) = p -1 • post(a p v i ) = p • prv(a p v i )[v i+1 ] = n/2 , if v i = v m-1 • prv(a p v i )[v j ] = u, ∀v j ∈ M \ {v i+1 }.
For each n ∈ N, OnePrv_n is of the class SAS-PUSC 0 . It can be solved by both P and P.

For each v i ∈ M, the longest chain of actions is:

chain v i (0, n -1) = (A(v i ), ≺ pre ) = a 1 v i , a 2 v i , ..., a n-1 v i
OnePrv_n_Cycle consists in OnePrv_n but, for each v i ∈ M, there is the additional

operator o 0 v i : post(o 0 v i ) = p = 0, pre(o 0 v i ) = n -1.
The prevail-conditions of o 0 v i follow the same rules as the other operators of OnePrv_n. For each v i ∈ M, o 0 v i creates a cycle in G O (v i ), and it allows chain of actions from 0 to 0 for instance:

chain v i (0, n -1) ⊕ a 0 v i = chain v i (0, 0)
OnePrv_n_Cycle is of the class SAS-PUSC n , that is, according to Lemma 4.4.1 and Theorem 4.4 [6, p.76], if n ≥ 3, the minimal solution plans are at most Υ 2 , which implies only P is complete. If n < 2, SAS-PUSC 2 is equivalent to the class SAS-PUC S 2 , i.e. Υ 1 , therefore all problems of this class can be solved by both P and P. I now describe two scalable instances for these two problems. For OnePrv_n, I define one = (Φ one , s 0 , s ) with:

s 0 = v i = 0 | v i ∈ M , s = v i = n -1 | v i ∈ M
For OnePrv_n_Cycle, I define oCy = (Φ oCy , s 0 , s ) with:

s 0 = v i = 0 | v i ∈ M , s = v 0 = n -1 ⊕ v i = 0 | v i ∈ M \ {v 0 }
Let consider the scalable problem instance oCy3 = (Φ oCy3 , s 0 = 0, 0, 0 , s = 1, 0, 0 ) of OnePrv_2_Cycle with m = 3. The operator set is described in Table 5.2.

In this problem, the minimal solution plan is:

∆ = a 1 v 2 , a 1 v 1 , a 1 v 0 , a 0 v 1 , a 0 v 2 .
Both planners, P and P, create 8 orders to solve the problem and return ∆, which is a minimal solution plan of 5 actions.

It can be proved that both P and P have to create at least (5 • m -7) orders to solve oCy of OnePrv_2_Cycle with respect to m, and they both return a minimal action plan ∆ of (2 • m -1) actions and that is Υ 1 .

O Tun3 pre() post() prv()

o 0 v 0 v 0 = 1 v 0 = 0 u, 1, u o 1 v 0 v 0 = 0 v 0 = 1 u, 1, u o 0 v 1 v 1 = 1 v 1 = 0 u, u, 1 o 1 v 1 v 1 = 0 v 1 = 1 u, u, 1 o 0 v 2 v 2 = 1 v 2 = 0 u, u, u o 1 v 2 v 2 = 0 v 2 = 1 u, u, u Table 5.2: Operator set of Φ oCy3 .

Empirical Comparison of Time Complexities (P vs P)

In this subsection, I empirically compare the time complexity of bothP and P. To do so, I used the two scalable problem instances mCy (MultiPrv_n_Cycle) and oCy (OnePrv_n_Cycle) described in subsection 5.1.1. As a reminder, the worst-case time complexity of P is O(n • m 2 ) (cf. Theorem 4.2 [6, p.90]), whereas the worst-case time complexity of P is

O(|A| + |E A |) ≡ O(n • m 2 ) (cf. Theorem 4.5.3).
With n being the number of values inside a variable domain, m the number of state variables in the problem, A the actions of the plans, |E A | the number of orders between the action of A. P is linear in n and quadratic in m while P is linear in the number of actions plus the number of orders between them, which, in very constrained problems, is equivalent to the complexity of P.

In the following, I present two experiments. In the first one (1), P and P have to solve MultiPrv_2_Cycle and OnePrv_2_Cycle in function of m. That is, n = 2 is fixed, while m ranges from 0 to 4000. In the second one (2), P and P have to solve MultiPrv_n_Cycle and OnePrv_n_Cycle in function of n and with three different fixed m. (1) and (2) consist in observing and comparing the runtime performance of both algorithms in function of m or n respectively.

Runtime performance comparison in function of m:

As explained previously, MultiPrv_2_Cycle and OnePrv_2_Cycle are both problems of the class SAS-PUSC 2 which means all their problem instances can be solved by both P and P. Hence, both algorithm can be compared with the problem instances mCy (MultiPrv_n_Cycle) and oCy (OnePrv_n_Cycle) presented in subsection 5.1.1.

In the following, I fixed the number of values to 2: n = 2. Figure 5.2 shows the runtime performance of both P (Green data set) and P (Red data set) solving the problem instances of MultiPrv_2_Cycle and OnePrv_2_Cycle respectively, and in function of an increasing number of state variables m. I will use the name of the problems, MultiPrv_2_Cycle and OnePrv_2_Cycle, to refer to the problem instances mCy and oCy with n = 2, hoping the reader will not be confused. Each green dot (resp. red dot) gives the average runtime of P (resp. P) to solve MultiPrv_2_Cycle or OnePrv_2_Cycle with a fixed m 15 times. For example, for m = 2002, the average runtime of P to solve MultiPrv_2_Cycle 15 times is 0.08ms. I previously explained that MultiPrv_2_Cycle is designed to be solved with ). This is why P solves OnePrv_2_Cycle in a linear manner: this problem is designed to have a non-linear minimal solution plan with a linearly increasing number of orders as a function of m. P then solves MultiPrv_2_Cycle in a quadratic manner: this problem is designed to have a non-linear minimal solution plan with a quadratically increasing number of orders as a function of m (Green data sets).

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •                                                                                 ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ • Data from   Data
• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • MultiPrv_2_Cycle
Figure 5.3 shows that P is at least 2 times faster than P to solve MultiPrv_2_Cycle, especially when m is big. When m is small (< 200), then P is even faster, up to a factor 10 when m = 102. Then, we can also see in Figure 5.3 that P is at least 65 times faster than P (m = 150) to solve OnePrv_2_Cycle. And the higher m, the higher the ratio. When m = 3900, P is 178 times faster than P. MultiPrv_2_Cycle has a very constrained structure and shows that the setup and the use of identifiers for P helps achieving better results. OnePrv_2_Cycle, on the contrary, is a less constrained and shows that the use of predecessor sets are relevant, especially for bigger problem instances, as it greatly enhances the performances. The solving of OnePrv_2_Cycle also shows that P can do a lot of useless work. It evaluates every prevail-condition of each action, which, in the case of OnePrv_2_Cycle, means P evaluates m -2 undefined prevail-conditions per action. It should finally be noticed that real-world problems are very unlikely to be as constrained as MultiPrv_2_Cycle. In fact, they are more likely to be low or medium constrained, which further strengthens my strategy of using a setup before planning.

Runtime performance comparison in function of n:

C. Bäckström explained in his thesis that he didn't succeed in empirically show the linearity of his algorithm in function of the size of the state variable domains n. He instead got a superlinear behavior and justified the result with some other experiments. He explained that the possible reason for the superlinearity was the use of Lisp and that "many details, like memory management, are out of control of the programmer" [6, p.92]. Contrary to Lisp, the C++ programming language allows memory management and the use of other tools, such as pointers, to counteract or avoid disruption from external events during data collection. With the help of my benchmarks OnePrv_n_Cycle and MultiPrv_n_Cycle, I was able to carry out experiments to prove the linearity in function of n of both P and P. The results are given in Figure 5.4. The red data refers to the performance of P while the green data refers to the performance of P. Figure 5.4a shows the experiment with MultiPrv_n_Cycle. Each linear curve shows the solving of MultiPrv_n_Cycle with a fixed m (m ∈ {2, 4, 8}) in function of n, by P (red curves) and by P (green curves). Figure 5.4b presents similar results but for the problem OnePrv_n_Cycle. These results corroborate that both P and P are linear in n.

Figure 5.5 presents the runtime performance ratio of P over P when solving MultiPrv_n_ Cycle (Figure 5.5a) or OnePrv_n_Cycle (Figure 5.5b), with fixed m and in function of n. Overall, my algorithm (P) was faster than C. Bäckström's one (P) in these experiments. In both subfigures of Figure 5.5, P is several hundred times faster than P when n is small. Then, the higher n is, the more the ratio tends towards 10.

It should be reminded that both OnePrv_n_Cycle and MultiPrv_n_Cycle are of the class SAS-PUSC n , which implies only P is complete to solve them. Because it exists instances that are Υ 2 (Theorem 4.4 [6, p.76] and Lemma 4.4.1). However, the scalable instances oCy and mCy used in the experiments and described in subsection 5.1.1 are Υ 1 , i.e. the minimal solution plan of these instances is always Υ 1 . Therefore, these instances are solvable by P. In addition to that, the solution plans of P were correct as they were equal to those of P.

Another Scalable Example: BlockWorld

The BlockWorld is a very well-known planning problem [41]. C. Bäckström has created a SAS-PUBS version of it in his thesis [6, p.93] with 3 blocks. It can be generalized as follows:

M = {X 1 onX 2 , ..., X 1 onX k , ..., X k onX k-1 },
where k ∈ N is the number of blocks and X k the k th blocks. There are m = |M| = k • (k -1) state variables. Let consider the two blocks X and Y such as X = Y , there are two operators named MoveXfromYtoTable and MoveXfromTabletoY and define as:

• MoveXfromYtoTable -post(MoveXfromYtoTable)[XonY ] = 0 -pre(MoveXfromYtoTable)[XonY ] = 1 -prv(MoveXfromYtoTable) = u | ∀v ∈ M , then: * ∀C ∈ Blocks \ {X, Y }, prv(MoveXfromYtoTable)[XonC] = 0 * ∀C ∈ Blocks \ {X}, prv(MoveXfromYtoTable)[ConX] = 0 * ∀C ∈ Blocks \ {X, Y }, prv(MoveXfromYtoTable)[ConY ] = 0 • MoveXfromTabletoY -post(MoveXfromTabletoY)[XonY ] = 1 -pre(MoveXfromTabletoY)[XonY ] = 0 -prv(MoveXfromTabletoY) = prv(MoveXfromYtoTable)
The following s 0 and s * form the reverse tower scalable example. That is, s 0 represents a tower of k ∈ N blocks s.t. ∀1 < i ≤ k, the block X i is on top of X i-1 (With 3 blocks, cf. (a) in Fig 5 .6), and s * represents the reverse tower: (a) MultiPrv_n_Cycle.

∀1 ≤ i < k, the block X i-1 is on top of X i (With 3 blocks, cf. (b) in Fig 5.6). • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •                                                                                                                                                                                                                                                                                                             • Data from   Data from ℙ
• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •   
• • • • • • • • • • • • • • • • •• • • •• • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • •• • •• • • • •• •• • • • • •• • • • •• •• • •• ••• • ■ ■ ■ ■ ■ ■ ■■ ■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■■ ■ ■ ■ ■ ■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■■ ■ ■ ■■■■■■ ■■ ■■ ■ ■ ■ ■ ■ ■■ ■ ■ ■ ■■ ■ ■ ■ ■■■ ■ ■■ ■ ■■■■
▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲ ▲ ▲▲ ▲ ▲▲ ▲ ▲▲ ▲ ▲▲▲ ▲▲▲▲ ▲ ▲ ▲▲ ▲▲▲▲▲▲▲▲ ▲▲ ▲ ▲▲▲ ▲▲▲ ▲▲▲▲▲▲▲ ▲ ▲▲ ▲ ▲▲▲▲▲ ▲ ▲▲▲▲▲▲▲▲ ▲ ▲ ▲ ▲▲▲ ▲ ▲ ▲ ▲▲▲ ▲▲▲ (a) MultiPrv_n_Cycle.

• • • • • • •• • • • •• • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • •• • • • ••• • • • •• • • • •• • • •• •• •• • • • •• ••••• • • ■ ■ ■ ■ ■ ■ ■ ■ ■ ■■ ■ ■ ■ ■ ■ ■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■■■ ■■■ ■ ■■ ■ ■ ■■ ■ ■ ■ ■ ■■ ■ ■■ ■ ■■ ■■■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■■ ■ ■ ■ ■ ■ ■■■■■■■■■■ ■ ■ ■■ ■ ■■ ■ ■■
▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲▲ ▲ ▲ ▲▲▲ ▲ ▲▲ ▲▲▲ ▲▲ ▲▲ ▲▲▲ ▲▲ ▲ ▲▲▲▲▲▲ ▲▲▲▲▲▲▲▲ ▲ ▲▲▲▲ ▲▲▲▲ ▲▲▲▲▲▲ ▲ ▲▲▲▲ ▲ ▲▲▲▲▲▲▲ ▲ ▲ ▲▲ ▲▲▲▲▲

• ∀1 < i ≤ k, s 0 [X i onX i-1 ] = 1.
All the other state variables are set to 0.

• ∀1 ≤ i < k, s * [X i-1 onX i ] = 1.
All the other state variables are set to 0.

BlockWorld with 3 blocks: In this paragraph, I illustrate the SAS-PUBS representation of the BlockWorld with 3 blocks. Let Φ 3blocks = (M, S, O) a SAS-structure of the BlockWorld with 3 blocks, and rev = (Φ 3blocks , s 0 = 0, 0, 1, 0, 0, 1 , s = 1, 0, 0, 1, 0, 0 ) the reverse tower problem presented on Fig. 5.6. The operator set O of Φ 3blocks is presented in Tab. 5.3. Table 5.4 shows that P is extremely fast to solve rev , it solved rev with 50 blocks in 88.6µs. The number of blocks, however, ranges from 0 to 50 and it is difficult to add more. The reason is that the number of state variables grows in a quadratic manner in function of the number of blocks (m = |M| = k • (k -1)) with this representation. It is therefore not possible to consider ten thousand of blocks [42]. With that amount of block, one SAS-PUC 2 operator would at least take O(m 2 ) = 10 3 To. Eventually, although P is extremely fast, the representation takes a lot of space.

O pre() post() prv() Operator Name o 0 v 0 v 0 = 1 v 0 = 0 u, 0, 0, u, 0, 0 AfromBtoT o 1 v 0 v 0 = 0 v 1 = 0 u, 0, 0, u, 0, 0 AfromTtoB o 0 v 1 v 1 = 1 v 1 = 0 0, u, 0, 0, 0, u AfromCtoT o 1 v 1 v 1 = 0 v 1 = 1 0, u, 0, 0, 0, u AfromTtoC o 0 v 2 v 2 = 1 v 2 = 0 0, u, u, 0, 0, 0 BfromAtoT o 1 v 2 v 2 = 0 v 2 = 1 0, u, u, 0, 0, 0 BfromTtoA o 0 v 3 v 3 = 1 v 3 = 0 0, 0, 0, u, u, 0 BfromCtoT o 1 v 3 v 3 = 0 v 3 = 1 0, 0, 0, u, u, 0 BfromTtoC o 0 v 4 v 4 = 1 v 4 = 0 u, 0, 0, 0, u, 0 CfromAtoT o 1 v 4 v 4 = 0 v 4 = 1 u, 0, 0, 0, u, 0 CfromTtoA o 0 v 5 v 5 = 1 v 5 = 0 0, 0, u, 0, 0, u CfromBtoT o 1 v 5 v 5 = 0 v 5 = 1 0, 0, u, 0, 0, u

Realistic Benchmarks

As I aim to provide a real-time planning algorithm it is crucial to test the C++ implementation of my planner P with realistic benchmarks. First to show that it is possible to represent practical problems with my class of problems, and second to prove that my algorithm satisfies the real-time constraint.

For this purpose, I introduce 3 practical benchmarks that I created inspired by the following commercial video games: Red Dead Redemption 2 (RDR-2) [11], Assassin's Creed: Origins (ACO) [12] and Horizon Zero Dawn (HZD) [13]. I name the benchmarks after those of the games. I present the world representation, some operators and some goals for each game. I also introduce operator graphs for some NPCs of each game so as to show my structural restrictions C 2 and C * 2 in practice. Finally, each benchmark has its specificity: the RDR-2 model has several different types of NPCs, each with its own representation of the world ; the ACO model has a base-NPC and a specialized one that inherits the former. The operators in ACO have parameters1 that allow contextualization ; the HZD model has a single type of NPC with a single goal. This model shows how to represent an NPC having a single goal with my class of problem. The section is organized as followed: I introduce RDR-2, ACO and HZD; Then, I present the experiments I carried out to study the performances of both TP and MPUS. I finally explain the results, suggest a discussion and give a conclusion.

O Pre Post Prevail v 0 , v 1 , v 2 Explanation o 0 v 0 v 0 = 1 v 0 = 0 u, u, u Drop Haystack o 1 v 0 v 0 = 0 v 0 = 1 u, 0, u Take Haystack o 2 v 0 v 0 = 1 v 0 = 2 u, u, u Fill Feeder o 0 v 1 v 1 = 1 v 1 = 0 u, u, u Drop Bucket o 1 v 1 v 1 = 0 v 1 = 1 0, u, u Pick up Bucket o 1 v 2 v 2 = 0 v 2 = 1 u, 1, u Fill Bucket with Water o 2 v 2 v 2 = 1 v 2 = 2 u, 1, u Fill Horse Trough v 0 : HayStack, D v 0 = {0 : none, 1 : inHands, 2 : inF eeder}. v 1 : Bucket, D v 2 = {0 : none, 1 : inHands}. v 2 : W ater, D v 2 = {0 : bucketEmpty, 1 : bucketF illed, 2 : inT rough}.

Read Dead Redemption 2

Red Dead Redemption 2 (RDR-2) was released by Rockstar Studios in October 2018 [11]. It is a action-adventure and open-world video game that takes place in Southern United States in the late Wild West era, in 1899. The player plays as Arthur Morgan, an outlaw member of the Van Der Linde gang who must deal with the decline of the Wild West whilst attempting to survive several different adversaries like the government forces, the rival gangs, etc. This very successful video game [43] has NPCs of great quality which improves credibility and dynamism of the virtual world of RDR-2. Christopher Livingston, authors of articles for PC Gamer, wrote the following about the NPCs of RDR-2 [44]:

The most compelling open worlds are the ones that feel like they would exist even if you never visited them, and the key to that illusion is to fill the world with convincing people. I'm not talking about the main characters, the questgivers, or the speech-makers. I'm talking about everyone else -the background extras that fill the gaps in the world, the ones you tend to barely notice unless you slow down and start paying attention to them.

Videos of NPCs' daily life can be found on-line and show the details of their behaviors. At the time stamp 5:36 of the video [45], we follow a Horse Breeder that is off to work. He goes to feed his horses by filling the water trough first and then the feeder.

Table 5.5 presents a possible SAS-PUC * 2 operator set for the Horse Breeder. Similar to the video, my version of the Horse Breeder can interact with a bucket and the haystacks in order to fill the water trough and the feeder for his horses. Then, his work goal is to feed the horses: s (Feed the horses) = 2, 0, 2

The main difficulty in this representation is to ensure the NPCs do not carry a haystack and the bucket at the same time as it is physically demanding. Hence the prevail-conditions: prv(Haystack)[v 1 ] = 0 and prv(Bucket)[v 0 ] = 0. With the goal s (Feed the horses), it is not possible for the Horse Breeder to fill the feeder first, however. The planner will necessary return a plan whose first subgoal is to fill the water trough then the feeder.

Because the action Fill Horse Feeder sets (v 0 = 2) permanently, which means the prevailcondition of Take Bucket can never be satisfied after Fill Horse Feeder while planning. If we want to fill the feeder first anyway, then we need two subgoals:

• s (Fill feeder) = 2, 0, s 0 [v 2 ] • s (Fill trough) = s 0 [v 0 ], 1, 2
Once the Horse Breeder has finished the goal s (Fill feeder), then he no longer carries an haystack. Therefore, the achievement of s (Fill feeder) set v 0 to the value 0. Same for s (Fill trough), once achieved v 2 can be reset to 0 for the Horse Breeder. It works the same way with s (Feed the horses). In a way, the state variables HayStack and W ater can be seen as targetable objects. Once in the feeder, the haystack is no longer targetable. That is, the Horse Breeder has to target another one if he wants to repeat the process. This is the same thing for the water, once in the water trough, it is no longer usable and the Horse Breeder needs to find a water source to refill the bucket. Hence, if the Horse Breeder calls the planner again to s (Feed the horses) with new targeted items, then s 0 [v 0 ] = 0 and s 0 [v 2 ] = 0 necessarily. Figures 5.7 and 5.8 respectively shows the domain operator graphs and the full operator graph of the Horse breeder. What is interesting with this Horse Breeder representation is that it highlights post-uniqueness does not imply pre-uniqueness. The operators Drop Haystack and Fill Horse Feeder are post-unique but they share the same precondition: pre(Drop Haystack)[v 0 ] = pre(FillHorseFeeder)[v 0 ] = 1, i.e. they both require the Haystack to be inHands (1). It results a domain operator graph with an unusual shape (cf. Figure 5.7), that is a graph with an unique cycle that does not contain all the nodes. The shape is unusual because, most of the time, when representing a problem, the resulting domain operator graphs are either a directed tree (cf. appendices B: id=2 B. Similar to the Horse Breeder, I have designed 6 more NPC types: the Farrier, whose goals are s (Soeing the horse) and s (Forge horseshoes); the Native hunter, whose goal is to s (Hunt); the Bandit, whose goal is to s (Rob a bank); the Innkeeper, whose goals are to s (Serve a drink) and s (Assing a room); the Dancer, whose goal is to s (Dance); and, finally, the Drunk Citizen, whose goal is to s (Forget). The SAS representation of these NPCs can be found in appendix F.

Assassin's Creed: Origins

Assassin's Creed Origins (ACO) is action role playing video game developed by Ubisoft Montreal and released in October 2017 [12]. The tenth installment of the Assassin's Creed serie sets in Egypt, near the end of the Ptolemaic period (49-43 BC), where the player plays as a Medjay named Bayek of Siwa. In order to make their magnificent open-world feel more alive and believable, the developers went in great detail with their NPCs, giving them autonomy and freedom through a Goal-Oriented AI system [46]. Soldiers, citizens, and even the fauna, have goals and needs that the player can witness or feel while playing [9]. At night, for example, the guards are less present because they are eating or resting. This allows the player to draw tactics based on the habits of the NPCs.

As there are a lot of different types of citizens in ACO, I have decided to create a base-NPC named Citizen. Then I introduce the Armorer, a citizen whose work is to create swords 2 because none of the "Green" nodes are linked to the "Darker Green" one when the "Yellow" nodes are removed (cf. Figure 5.9b).

Citizen

O Pre Post Prevail v i |i ∈ [[0, 6]] Explanation o 1 v 0 v 0 = 0 v 0 = 1 u, u, u, u, u, u, 0 Buy Food(?pos) o 2 v 0 v 0 = 1 v 0 = 2 u, u, u, u, u, u, u Eat o 1 v 1 v 1 = 0 v 1 = 1 u, u, u, u, u, u, 0 Buy Drink(?pos) o 2 v 1 v 1 = 1 v 1 = 2 u, u, u, u, u, u, u Drink o 0 v 2 v 2 = 1 v 2 = 0 u, u, u, 0, u, u, 0 Store (?item,?pos) o 1 v 2 v 2 = 0 v 2 = 1
u, u, u, 0, u, u, 0 Take (?item,?pos)

o 0 v 3 v 3 = 1 v 3 = 0 u, u, u, u, u, u, u Stop Wandering o 1 v 3 v 3 = 0 v 3 = 1 u, 2, u, u, u, u, 0 Have a Walk o 0 v 4 v 4 = 1 v 4 = 0 u, u, u, u, u, u, u Stop Debate o 1 v 4 v 4 = 0 v 4 = 1
u, u, u, u, u, u, u Debate(?with, ?pos) [7,10]]}. and sabers. We have given our citizen the possibility to eat, drink, buy some food, wander, debate and work (Table 5.6). Although our operator representation is quite generic, I have added some parameters to some of them (Buy Food, Buy Drink, Store, Take, Debate, Rest and Work) so as to allow some contextualization. For instance, if the NPC is a musician who like to play in the street, then we can contextualize: Work(?Luth, ?MarketPlace). We can also represent a group of musicians: it is merely a set of NPCs playing different instruments at the same place:

o 1 v 5 v 5 = 0 v 5 = 1 u, u, u, 0, 0, u, 0 Rest (?pos) o 0 v 6 v 6 = 1 v 6 = 0 u, u, u, u, u, u, u End Work o 1 v 6 v 6 = 0 v 6 = 1 2, 2, 1, 0, u, u, u Work (?item, ?pos) v 0 : Hunger, D v 0 = {0 : noF ood, 1 : hasF ood, 2 : f ed}. v 1 : T hirst, D v 1 = {0 : noDrink, 1 : hasDrink, 2 : hydrated}. v 2 : hasItem, D v 2 = {0 : f alse, 1 : true}. v 3 : W andering, D v 3 = {0 : f alse, 1 : true}.
. M Citizen = {v i | i ∈ [[0, 6]]} is the citizen state variable set. Armorer A Pre Post Prevail v i |i ∈ [[0, 10]] Explanation o v v 7 = 0 v 7 = 1 u, u, u, u, u, u, 1, u, u, u, u Put Metal in Foundry o v v 8 = 0 v 8 = 1 u, u, u, u, u, u, 1, u, u, u, u Collect Smelted Metal o v v 9 = 0 v 9 = 1 u, u, u, u, u, u, 1, u, 1, u, u Forge a Sword o 1 v 10 v 10 = 0 v 10 = 1 u, u, u, u, u, u, 1, u, 1, u, u
M Armorer = M Citizen ∩ {v j | j ∈ [
• Work NPC 1 (?Luth,?MarketPlace)

• Work NPC 2 (?Drum,?MarketPlace)

• Work NPC 3 (?Triangle,?MarketPlace)

• Work NPC 4 (?Voice,?MarketPlace) All these musicians must take their instrument as well, so:

• Take NPC 1 (?Luth,?npc1HomeLuthStand)

• Take NPC 2 (?Drum,?npc2HomeChest)

• Take NPC 3 (?Triangle,?npc3HomeChest)

• NPC 4 always has is voice, so his hasItem(v 2 ) state variable is always true I have added these parameters in order to illustrate that it is possible to contextualize an operator without modifying the overall representation. Furthermore, I wanted to point out that MoveTo operators must not be represented in any SAS-PU representation. MoveTo operators are related to pathfinding, representing them in any SAS-PU representation is almost impossible as it is very likely to not respect the post-uniqueness. A position to reach can merely be added as a parameter. Indeed, almost every action is preceded by a MoveTo, so it can implicitly be added inside the animation of the action.

Table 5.7 is a more specific Citizen: the Armorer. The Armorer is a Citizen, so it inherits the operators presented in Table 5.6. Plus, he has more detailed operators to accomplished its duty, namely: Put Metal in Foundry, Collect Smelted Metal, Forge a Sword, Forge a Saber. As these operators have to be performed while working, they all have the prevail-condition (prv): (v 6 = 1). Then, it can be noticed that the Armorer can debate while working, but not have a walk. Debate has no specific prevail-conditions, but Have a Walk requires the work in progress to be stopped (prv(Have a Walk)[v 6 ] = 0).

Let now describe some possible goal states for the citizen2 :

1. s (Feed) = v 0 = 2, v 1 = 2, v 6 = 0 ⊕ v i = s 0 [v i ] | v i ∈ M \ {v 0 , v 1 , v 6 } 2. s (Rest) = v 3 = 0, 0, 1, 0 ⊕ v i = s 0 [v i ] | v i ∈ M \ {v 3 , v 4 , v 5 , v 6 } 3. s (Debate) = v 4 = 1 ⊕ v i = s 0 [v i ] | v i ∈ M \ {v 4 } 4. s (Wander) = v 1 = 2, v 3 = 1, v 6 = 0 ⊕ v i = s 0 [v i ] | v i ∈ M \ {v 1 , v 3 , v 6 } 5. s (Work) = v 0 = 2, 2, 1, 0, v 6 = 1 ⊕ v i = s 0 [v i ] | M \ {v 0 , v 1 , v 2 , v 3 , v 6 } 6. s (ForgeSword) = s (Work) ⊕ v 7 = 1, v 8 = 1, v 9 = 1 7. s (ForgeSaber) = s (Work) ⊕ v 7 = 1, v 8 = 1, v 10 = 1 8. s (WanderAndRest)) = v 1 = 2, v 3 = 1, v 5 = 1 ⊕ v i = s 0 [v i ] | v i ∈ M \ {v 1 , v 3 , v 5 } 9. s (Eat,Debate,Rest)) = v 0 = 2, v 4 = 1, v 5 = 1 ⊕ v i = s 0 [v i ] | v i ∈ M \ {v 0 , v 4 , v 5 } 10. s (BuySupplies)) = v 0 = 1, v 2 = 1 ⊕ v i = s 0 [v i ] | v i ∈ M \ {v 0 , v 1 }
The description of the goals as well as some prevail-conditions are very precise because of SAS in which restriction (S7) imposes that s 0 and s must be totally defined. It may exists way to automate this, however. For example, the goal s (Work) could first be instantiated with the start state, then each required values on specific state variables could be found and then defined with the main operator, namely Work (a 6 v 1 ), by looking at its prevail-conditions and its post-condition:

s (Work) = s 0 , ∀v i ∈ M s.t. prv(Work)[v i ] = u, s (Work)[v i ] = prv(Work)[v i ],
s (Work)[v 6 ] = post(Work). Let imagine an idle armorer that is hungry, thirsty and tired, we have: s Armorer 0 = 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 This is the beginning of the day and he has to work, however. And we also want him to have rested and tidied up his belongings at the end of the day, we can give him the following goal:

s Armorer = 2, 2, s 0 [v 2 ], s 0 [v 3 ], s 0 [v 4 ], s 0 [v 5 ], s 0 [v 6 ], 1, 1, 1, 1
s Armorer expresses that he has fed (v 0 = 2) and hydrated (v 1 = 2) himself during the day, that he has forged sabers and swords (v 9 = 1, v 10 = 1) using his foundry (v 8 = 1, v 9 = 1), and that he is now doing what he was doing at the beginning of the day, namely idling

(v i = s 0 [v i ], ∀i ∈ [[2, 6]]
). If we also want our armorer to rest, we can add to the goal: s [v 5 ] = 1. Which will give the opportunity for our NPC to rest.

With our topological planning algorithm, the minimal solution plan is: ∆ = Buy Food(?pos), Eat, Buy Drink(?pos), Drink, Rest (?pos), Take (?item, ?pos), Work (?item, ?pos), Put Metal in Foundry, Collect Smelted Metal, Forge a Sword, Forge a Saber, End work, Store(?item, ?pos)

Concerning the parameters item and pos, our armorer works at a blacksmith and he needs a hammer that could be stored in a chest. We can thus contextualize the following operators: Take (?hammer, ?armorerChest); Store (?hammer, ?armorerChest); Work(?hammer, ?blacksmith). He can buy food and drinks at the shop, thus: Buy Food (?shop), Buy Drink(?shop). And finally, he can rest at many places, such as his bedroom, stairs, benches, etc. Thus, if there is a staircase while going to work, we can have: Rest(?stairs). It should be noticed that the contextualization of an action must not be done while planning. It must either be update before planning thanks to dedicated variables, or it can be evaluated during the execution of the action. To contextualize an action while planning is burdening the planner and threatens the real-time constraint.

Horizon Zero Dawn

Horizon Zero Dawn (HZD) is an action role play game developed by Guerrilla Games and released in early 2017 [13]. The player plays a huntress named Aloy in a world full of high-tech colossal machines that have appeared at the fall of human civilization. These machines are categorized in classes: the acquisition class, whose role is to harvest resources; the combat class, whose role is to guard vulnerable machines from hunters; the recon class, whose initial role was to search for suitable terraforming land, they finally serve as guards and lookouts in the game; the transport class, whose role is to help acquisition machines to move large amount of resources. There are other types of machines whose description can be found in [47]. In the following, I present a possible SAS-PUSC 0 representation for acquisition machines.

The acquisition machines, which I henceforth denote A-M, harvest resources of the HZD world. Their role is crucial as they assist the terraforming program and allow for the further construction of machines. This description, however, is more folkloric than anything else as this is not what really happens in-game. Although there exist stunning combat scenes in HZD, which are mainly due to great animations, the overall behavior of the machines can be disappointing to those who seek consistency with the life goal of these machines. Through planning, however, it is possible to create operator sets and represent life goals to fit the description of these machines.

Let consider the following SAS-PUSC 0 problem that models an A-M. It has a unique goal: Harvest. An A-M repeats this goal each time it is completed, or reevaluates it each time the on-going plan is invalidated by one or more disruptions. For each A-M, we can model a harvest routine and some disruptions with the following state variable descriptions:

• v 0 : oreDepositF ound, D v 0 = {0 : f alse, 1 : true}. • v 1 : oreDepositP osShared, D v 1 = {0 : f alse, 1 : true}. • v 2 : harvestRoutine, D v 2 = {0 : none, 1 : oreHarvested, 2 : oreRef ined, 3 : ref inedOreStored}. • v 3 : isCharged, D v 3 = {0 : f alse, 1 : true}. • v 4 : system, D v 4 = {0 : damaged, 1 : operational}. • v 5 : disturbanceExist, D v 5 = {0 : f alse, 1 : true}. • v 6 : isT hreatened, D v 6 = {0 : f alse, 1 : true}.
v 0 checks whether the A-M is surrounded by ore deposits. If not, the A-M can Scan for Ore Deposit, which allows it to find points of interest. I wanted the position of ore deposits to be a shared information between A-M. Thus, once the action Scan for Ore Deposit is over and succeeded, or if at least one A-M knows where there are some deposits, it can Share Ore Position to other machines. This operator affects the state variable v 1 . As we cannot share a position without knowing it, (v 0 = 1) is a prevail-condition for Scan for Ore Deposit. Then, v 2 takes care of the harvest routine. At the beginning of a scene, or when starting the game, v 2 is initially set to 0. It means two things: first the harvest routine

has not yet started, and second the machine does not have any resource. v 2 will be reset to 0 each time the subgoal (v 2 = 3) is accomplished, that is, each time an A-M succeeds to store a refined ore. Otherwise, v 2 keeps its value. It means that if the harvest routine is disturbed, the machine does not restart the on-going routine, but resume it instead once all the disruptions have been taken care of. Let imagine an A-M heading towards a storage to Store Refined Ore, then v 2 = 2. If the player attacks the A-M at this point, then the A-M has to reevaluate the situation with s 0 [v 2 ] = 2 (and s 0 [v 6 ] = 1 because it is under attack). Let now consider the A-M has accomplished its goal s [v 2 ] = 3, then the A-M has to reevaluate the situation with s 0 [v 2 ] = 0 because it has to restart the whole harvest routine. I also give A-M some sensing state variables: v 3 , v 4 , v 5 and v 6 . v 3 and v 4 are internal sensors that respectively check whether the system is charged or damaged. v 5 and v 6 sense the environment, updating their value if there is any disturbance or threat. To react to these disruptions, the A-M can: Rest, Repair, Scan towards Disturbance and Attack. These actions respectively affect v 3 , v 4 , v 5 and v 6 . Tab. 5.8 gives the full representation of these operators.

Acquisition Machines (A-Ms

) O Pre Post Prevail v i |i ∈ [[0, 6]] Explanation o 1 v 0 v 0 = 0 v 0 = 1 u, u, u, 1, 1, 0, 0 Scan for Ore Deposit(?fromPos) o 1 v 1 v 1 = 0 v 1 = 1 1, u, u, u, u, 0, 0 Share Ore Position(?listOrePos) o 1 v 2 v 2 = 0 v 2 = 1 1, u, u, 1, 1, 0, 0 Harvest Ore(?oreType,?orePos) o 2 v 2 v 2 = 1 v 2 = 2 u, u, u, 1, 1, 0, 0 Refine Ore o 0 v 2 v 2 = 2 v 2 = 3 u, u, u, 1, 1, 0, 0 Store Refined Ore(?storePos) o 1 v 3 v 3 = 0 v 3 = 1 u, u, u, u, u, 0, 0 Rest(?pos) o 1 v 4 v 4 = 0 v 4 = 1 u, u, u, 1, u, 0, 0 Repair o 0 v 5 v 5 = 1 v 5 = 0 u, u, u, u, u, u, 0 Scan towards Disturbance o 0 v 6 v 6 = 1 v 6 = 0 u, u, u, u, u, u, u Attack
Table 5.8: (Based on HZD) Operator representation for the Acquisition Machines (A-Ms). 6]]} is the state variable set of the A-Ms.

M AM = {v i | i ∈ [[0,
There are two main points in this representation. First, an A-M must take care of all existing disruptions before doing anything else. That is, they have to be charged, operational, not threatened and not disturbed to perform its routine actions. Hence the prevail-conditions of Harvest Ore, Refine Ore, Store Refined Ore and Scan for Ore Deposits in Tab. 5.8. These prevail-conditions ensure that an A-M will not harvest resources while under attack, for instance. That, instead, it will Attack first because (v 6 = 0) is required by these actions. Second, there is an order to the treatment of the disruptions. (v 6 = 1) is more serious than (v 5 = 1), which is more serious than (v 3 = 0), which, finally, is more serious than (v 4 = 0). This results in the prevail-conditions of the operators Attack, Scan towards Disturbance, Rest, Repair presented in Tab. 5.8. As Preturns a totally ordered plan thanks to a topological sort, this representation ensures that Pwill only return relevant topological sorts as plans. For instance, the operator Attack solves the worst disruption, (v 6 = 1) into (v 6 = 0). As all other operators have (v 6 = 0) as a prevail-condition, Pwill have no choice but to return an action plan whose first action is Attack. On the contrary, Repair solves the least worst disruption, hence, if the other disruptions are on, the prevailconditions of Repair ensure that Attack, Scan towards Disturbance and Rest will be first sorted by P. This way of writing operators is a knowledge representation, if we want to ensure that our planner returns a plan whose first action is to Attack when threatened, then (v 6 = 0) must necessary be a prevail condition for all the other operators that may be instantiated while planning.

Remark: Concerning the eyes coloring in HZD, it is possible to represent the change of eye color with operators. I personally consider it as an animation that can be triggered when some state variables are being updated to specific values. As an example, if v 6 is updated to 1, the A-M is threatened so its eyes can turn red. If the system is disturbed (v 5 = 1) and not threatened (v 6 = 0), then the A-M can turn its eyes in yellow. In any other cases, its eyes are blue. This way of describing dependencies between actions creates a very constrained problem. This can be highlighted with Figure 5.14 which shows the complete operator graph of my SAS-PUSC 0 version of HZD. The blue directed edges represent the post-pre dependencies while the green directed edges represent the post-prevail dependencies. As there is only one goal, however, it is necessary to constraint the problem so as to ensure a pertinent totally ordered plan by the planner.

Eventually, this HZD version is a SAS-PUSC 0 problem. It is obviously (P), (U), and (S). The new structural restriction, C k with k = 0, means no requestable operator (Definitions 4.4.1 and 4.1.4) is in the cycle of its domain operator graph. This is the case in this HZD version because there is no cycle in any of the domain operator graph of the problem, as it can be seen in Figure 5.15.

Now that the world and some operators for the A-Ms have been modeled, the goal Harvest and some possible start states can be defined.

s (Harvest) = 1, 1, 3, 1, 1, 0, 0
This goal means that the A-M is charged, operational and safe, it knows where there are ore deposits and the positions are shared with its counterparts. Plus, it has completed its harvest routine: v 2 = 3. In this representation, the A-Ms will always try to reach this goal. So let's create some start variations so that the planner can return different plans: • s 0 (1) = 0, 0, 0, 0, 0, 1, 1 , this is the worst scenario, the A-M is threatened, disturbed, damaged, not charged, has not started its harvest routine and does not have any clue where ore deposits are.

• s 0 (2) = 1, 1, 0, 0, 0, 1, 1 , Same situation but this time the A-M knows where to start harvesting.

• s 0 (3) = 1, 1, 0, 1, 1, 0, 0 , the A-M can safely start to Harvest Ore as all its prevailconditions are satisfied.

• s 0 (4) = 1, 1, 2, 0, 1, 0, 0 , the A-M is low on energy, it needs to rest before resuming its harvest routine.

• s 0 (5) = 0, 0, 0, 1, 1, 0, 0 , the A-M is safe and operational, it can harvest ores but it first needs to Scan for Ore Deposits. It also has to Share Ore Position at some point.

• s 0 (6) = 1, 1, 1, 1, 1, 1, 1 , the A-M can start to refine the ore it has already harvested, but it is actually threatened and disturbed, so it has to manage the disruptions first.

Runtime Performances

Now that the realistic benchmarks have been introduced, I will use them to carry out experiments, presented in the first paragraph of this subsection, that aim to show and compare the performances of the algorithms P and P. In a second paragraph, I show the results of the experiments and then discuss the performances of both algorithms. In the third paragraph, I go deeper in the analysis of the performance of P by redoing an experiment with the SAS-PUC * 2 version of ACO and by using the software AMD µProf to perform a CPU analysis. I finally discuss and conclude the results. Description of the experiments Even though my algorithm P is theoretically linear in the number of operators plus the number of orders between actions, it does not imply the real-time constraint is respected in practice. As I aim to provide a planner capable of giving as many plans as possible in a very short amount of time, my C++ implementation of P must be tested accordingly. The question is: "How many NPCs in RDR-2, ACO or in HZD can get a plan in real-time from my C++ implementation of P?" First of all, planning in real-time, in video games, means planning between two frames. Taking into account that only 10% of the processing budget is granted for AI system, planning in real-time for a 60 FPS3 video game means the planner has 1 60 * 0.1 = 1, 67ms of planning time. As a consequence, the runtime threshold I have set for the experiment is 1, 5ms. This is below 1.67ms and leaves a margin of error of 10%, or 0.17ms, in practice. For the experiment, I did not implement P nor the benchmarks described in subsections 5.2.1, 5.2.2 and 5.2.3 in their respective and original game as I do not have their source code nor the time to do it. Therefore, I have implemented the benchmarks RDR-2, ACO and HZD in a C++14 project with default settings for Microsoft Visual Studio 2019, in order to solve them with my C++ implementations of P. Some logs were displayed with the Command Prompt in Windows 10 whereas runtime data (and some other data) were collected on .txt files and then analyzed with Mathematica. Each C++ simulation has a specific configuration detailed below:

× × ×× × × × × × × ×× × × × × × × × ×× × × × × × × × × × × × × × × × × × × × × × × × × × × × × ×× × × ×× × ×× × × × × × ×× ×× × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × ×× × × × × × × × × ×× × ×××× ××× × × × × × × × × × × × ×× × × ×× × × × × × × × × × × × × × × × × × × × × × × × × × ×× × × × ×× × × × × × ×× × × × × × × × × × × × × × × × × × × × × × × × × × × ×                                                                                                 SAS-
• RDR-2: As described in the subsection 5.2.1 and in appendix F, there are 7 types of NPCs: the Horse Breeder, the Farrier, the Bandit, the Native Hunter, the Innkeeper, the Dancer and the Drunk Citizen. Overall, there are 30 state variables, 56 operators and the biggest state variable domain is D v 5 from the Farrier with 5 values. For the experiment, all NPCs start with s 0 = v i = 0 | v i ∈ M RDR-2 and try to reach one of their goals (Table F.8). P had to return a plan for each of the 9 goals successively and an increasing number of times, thus simulating the growth of the number of NPCs.

• ACO: There are two types of NPCs: the Citizen and the Armorer. Overall, there are 11 state variables, 18 operators and the biggest domains are D v 0 and D v 1 with 3 values. Each NPC starts with s 0 = v i = 0 | v i ∈ M ACO and try to reach one of the 9 goals described in subsection 5.2.2. P had to return a plan for each of the 9 goals successively and an increasing number of times.

• HZD: There is only the acquisition machines (A-Ms). Each A-M has 7 state variables, 9 operators and its biggest domain is D v 2 with 3 values. Compared to the NPCs of RDR-2 and ACO, the A-Ms only have one goal but 6 different starts described in subsection 5.2.3. P had to return a plan for each of the 6 starts successively and an increasing number of times.

Finally, with these configuration, I measure and collect the runtime it takes P to provide a plan to a growing number of NPCs in these C++ simulations of RDR-2, ACO and HZD. The result of these experiments are given in Figure 5.16. I have carried out the same experiment with P to test it and to compare its performances with those of P. However, P solves problems of the class SAS + -PUS. There are some common problems with those of SAS-PUC * 2 that can be solved by both algorithm, however. These problems are grouped into classes of problems like SAS-PUSC 0 and SAS-PUSC 2 Hence, HZD does not need to be reworked as it is SAS-PUSC 0 , RDR-2 and ACO, on the contrary, require some changes. RDR-2 is SAS-PUC * 2 because of the Horse Breeder and the Farrier problems. If we remove these two types of NPCs, then RDR-2 becomes SAS-PUSC 2 . Indeed, all the remaining NPC problems are either SAS-PUSC 0 (Dancer, Innkeeper) or SAS-PUSC 2 (Bandit, Drunk Citizen, Native hunter). This SAS-PUSC 2 version of RDR-2 has 20 state variables, 33 actions and the highest state variable domain is D v 27 from the Dancer with 4 values4 . Concerning ACO, the Citizen problem is SAS-PUSC 2 but the Armorer problem is SAS-PUC * 2 . I have reworked the Armorer to produce a SAS-PUSC 2 version of it, presented in Table 5.9. Concerning the goals of the Armorer, most of them remain unchanged (cf. subsection 5.2.2), but he no longer has s (Work) and:

• s (ForgeSword) = 2, 2, 1, 0, 0, 1, s 0 [v 6 ], 1, 1, 1, s 0 [v 10 ] • s (ForgeSaber) = 2, 2, 1, 0, 0, 1, s 0 [v 6 ], 1, 1, s 0 [v 9 ]1 Citizen O Pre Post Prevail v i |i ∈ [[0, 10]] Explanation o 1 v7 v 7 = 0 v 7 = 1 2, 2, 1, 0, 0, u, u, u, u, u, u Put Metal in Foundry o 1 v8 v 8 = 0 v 8 = 1 2, 2, 1, 0, 0, u, u, u, u, u, u Collect Smelted Metal o 1 v9 v 9 = 0 v 9 = 1 2, 2, 1, 0, 0, u, u, u, 1, u, u Forge a Sword o 1 v10 v 10 = 0 v 10 = 1 2, 2, 1, 0, 0, u, u, u, 1, u, u Forge a Saber v 7 : rawM etals, D v7 = {0 : stored, inF oundry(1)}. v 8 : smeltedM etal, D v8 = {0 : canBeCollected, 1 : collected}. v 9 : f orgedSword, D v9 = {0 : f alse, 1 : true}. v 10 : f orgedSaber, D v10 = {0 : f alse, 1 : true}.
Table 5.9: The SAS-PUSC 2 version of the Armorer of ACO. This version of the Armorer inherits the operators from the Citizen, except the operators Work and End Word. The state variable v 6 : W orking is still part of M Armorer but its value is set to 0 permanently to satisfy the v 6 prevail-condition of the inherited operators. In this version we consider the Armorer works as soon as the planner uses the operators presented in this table. The Citizen problem is SAS-PUSC 2 in its original form. This new version of ACO has 11 state variables, 18 operators and the biggest domains remain D v 0 and D v 1 with 3 values. Eventually, HZD and the new versions of RDR-2 and ACO are solvable by both P and P. The runtime performances of these two algorithm solving these benchmarks are shown in Figure 5.17 for P and in Figure 5.18 for P.

Finally, the runtime performances of P and P presented in Figures 5.16, 5.17 and 5.18 take into account the initialization runtime, the planning runtime and the runtime of the variables clearing for each problem instance. For example with the Horse Breeder, s 0 = 0, 0, 0 and s = 2, 0, 2 form an instance of the Horse Breeder problem. For the experiment, I have measure the time it takes to initialize this instance, how many times it takes P to solve it and how many times it takes to clear all the local variables. This measurement procedure is done for every single NPC to produce the data presented in this subsection.

Explanation of the performances of both P and P First of all, concerning the data symbols used in Figure 5.16 and in Figure 5.17, the square is for SAS-PUSC k problems (k = 0 or k = 2) whereas the cross is for SAS-PUC * 2 problems. HZD is a SAS-PUSC 0 problem in both experiments, the data shown in Figures 5.16 and 5.17 are the same. In Figure 5.16, the red and yellow crosses refer to the SAS-PUC * 2 version of RDR-2 and ACO respectively, whereas the red and yellow squares in Figure 5.17 refer to the SAS-PUSC 2 versions of RDR-2 and ACO. Finally, in Figure 5.18, I have used the black color for the performances of P to distinguish them from those of P: the triangle and dot symbols are for the SAS-PUSC 2 versions of RDR-2 and ACO respectively, the diamond is for HZD.

The linear growth of these curves is explained by the fact the worst-case time complexities of both P and P are not a function of the number of NPCs. Therefore, the slope of these curves is merely the ratio of the runtime growth to the number of NPCs growth. It Figure 5.17: The performance of P when solving HZD (blue square) and the SAS-PUSC 2 versions of RDR-2 and ACO (red and yellow squares resp.)

                                                                                                                                                                                                                                                              SAS-PUSC2
results that the slope highlights the difficulty of both algorithms to solve a problem. The steeper the slope, the more difficult the problem. From these explanations, we can observe that P and P do not have the same criteria of difficulty. For instance, HZD is the hardest problem for P whereas this is RDR-2 for P. Table 5.10 gives the causality of this difference in difficulty. Table 5.10 shows the linear regressions of the sets of data, where the argument x refers to the number of NPCs, shows the number of state variables (|M|) and the average number of orders between actions (avg(|E A |)) for each problem. The performances of P are sorted in descending order of avg(|E A |) in Table 5.10. It highlights the slope are correlated with the average number of orders 5 . It is consistent with the worst-case time complexity of P which is linear in the number of orders plus the number of actions:

O(|A| + |E A |).
In other words, the more orders, the more difficult for P. It explains why HZD is the most difficult problem for P despite being the one with the less state variables and with the less operators. It is in fact more constrained than the other two, and besides, it is directly linked to the number of the defined prevail-conditions shown in Tab 5.8. Concerning P, its performances are sorted in descending order of |M| in Table 5.10. It highlights that the difficulty to solve a problem is correlated with the number of state variable (|M|). This is also consistent with its worst-case time complexity, which is quadratic in the number of state variables. Hence, the more state variables, the more difficult for P.

• • • • • • •• • • • •• • • • • • • • • • •• • • • • • • •• • • • •• • • • • • •• • • • • • • • •• • • • •• •• • •• • • • • • •• • •• • • • •• •• • • • • •• •• • • •• • • • • • •• •
▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲ ▲ ▲▲ ▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲ ▲ ▲ ▲▲ ▲ ▲ ▲▲ ▲ ▲ ▲ ▲ ▲▲ ▲ ▲ ▲▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲▲ ▲ ▲ ▲ ▲ ▲

◆◆ ◆ ◆◆ ◆ ◆◆ ◆ ◆ ◆ ◆◆ ◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆ ◆◆ ◆ ◆◆ ◆◆ ◆◆ ◆ ◆◆ ◆ ◆ ◆◆ ◆ ◆◆ ◆ ◆◆ ◆ ◆ ◆ ◆ ◆ ◆◆◆ ◆◆ ◆◆ ◆◆◆ ◆ ◆◆ ◆◆ ◆◆ ◆ ◆ ◆ ◆ ◆ ◆ ◆ ◆ ◆◆ ◆◆ ◆ ◆◆ ◆ ◆◆ ◆ ◆ ◆ ◆ ◆ ◆◆◆ ◆ ◆◆ ◆ ◆

SAS-PUCS2 instances solved by 

▲ Red Dead Redemption 2

• Assassin's Creed: Origins Finally, the key result of these data is that my C++ implementation of P was able to provide a plan to 3.4 millions of NPCs in less than 1.5ms for each of the three realistic benchmarks, which is huge and definitely respect the real-time constraint. On the contrary, P was only able to provide a plan to a hundred NPCs for each benchmark in this same amount of time. It implies my C++ implementation of P is about 34.000 times faster than my C++ implementation of P. There are many reasons explaining the huge performance differences between P and P. The first one is the pre-processing, or setup, for P. I have setup a very rigorous memory management (malloc, memset, hashing table), and I have allocated and filled data structures as much in advance so as to not lose time while planning. This setup does not explain everything, however. I have implemented a setup version of P and the setup version has only speeded up the result by a factor of 3. Another reason is that P does less work than P while planning. During the setup of P, The neighborhood sets N pre , N prv are created and some space are allocated for N clob , which is the neighborhood set of threatening actions (Def. 3.2.5). By doing so, P only browses the defined preand prevail-conditions while planning, whereas P browses all the prevail-conditions, the undefined one included. It makes P do a lot of useless work, especially with realistic benchmarks where the prevail-conditions are more often undefined than defined. Even with the most constrained SAS + -PUS problem, which I have named Multi_Cycle_X, only half of all the prevail-conditions are defined, which means only half of the prevail checking is useful in P. CPU Analysis by Profiling P Given the impressive results of P, it is crucial to have an internal view of the processor while performing P. The experiments were performed with the following configuration: AMD Ryzen™ 7 2700X (8-Core) CPU (3.7GHz), 32Gb of RAM and Windows 10 (64 bits) (cf. Table 5.11 for more details); P was written in C++14 with default settings for Microsoft Visual Studio 20196 . To observe the internal performance of the processor, we have used the software AMD µProf [48]. AMD µProf is a software used to perform CPU, GPU, and power analysis of applications running on Window®, Linux®, FreeBSD® operating systems on AMD processors. The AMD Ryzen™ 7 2700X has a base clock of 3.7GHz with a boost of up to 4.3GHz, which corresponds to ≈ 4 • 10 9 clock cycles per second. There is thus approximately a clock cycle every quarter of a nanosecond. Yet, our linear time algorithm P is able to provide a plan of size 4 (on average) to 3.4 millions citizens in the SAS-PUC * 2 version of Assassin's Creed: Origins (cf. Figure 5.16). It approximately gives ≈ 1 3 • 10 -9 second to build a plan, on average. That is, a third of a nanosecond to build one action plan. Eventually, it means P builds on average one plan in 1.2 clock cycles with the AMD Ryzen™ 7 2700X. , on how the C++ implementation of P is processed during the experiment. This graph shows the number of Retired Instructions (RI) and the Instruction Per Cycle (IPC) wrt. the number of NPCs. The left x-axis is for the number of RI, the right x-axis is for the IPC, and the y-axis is the number of NPCs. The solid curves show the RI whereas the dashed curves show the IPC, both wrt. TopoPlan, the name of our algorithm; and wrt. SearchExtension, BuildChain and DFSTopo, functions (methods) used by TopoPlan and Init, the method used to initialize the start and the goal states for each problem instance. So, to observe how P is processed while solving ACO by the AMD Ryzen™ 7 2700X, I have carried out the following experiments with the AMD µProf software: I asked P to provide a plan to respectively 500,000, 1,000,000, 2,000,000, 3,000,000, 4,000,000 and 5,000,000 armorers and citizens in ACO. All at once respectively, i.e. I did not set any repetition to smooth the results. For each of the 6 numbers, I have profiled P with the AMD µProf software, using the Assess Performance profile type, in order to collect internal performance data from the AMD Ryzen™ 7 2700X. Figure 5.19 shows the internal performance, focusing on the number of retired instructions (RI, solid curves with solid dots) and the instructions per cycle (IPC, dashed curves with empty square symbols). The number of RI is the number of instructions retired from execution and the IPC is the average number of RI per CPU cycle.

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ 1 × 10 6 2 ×
Concerning the Figure 5.19, TopoPlan is the C++ name I have given to P. BuildChain, SearchExtension and DFSTopo are C++ methods used by TopoPlan to solve a given problem instance. Init is a method to initialize the start (s 0 ) and the goal (s ) of a problem instance. The x-axis is the number of NPCs, for each curve there is respectively a point at 500,000, 1,000,000, 2,000,000, 3,000,000, 4,000,000 and 5,000,000 NPCs. The left y-axis is for the RI whereas the right y-axis is for the IPC. The AMD µProf software does not count precisely how many instructions it takes the processor to execute a program but counts samples instead. Thus, the left y-axis of Figure 5.19 does not give the precise number of instructions but a sample count of it, hence the term "retired" in retired instructions.

The higher the IPC, the better. It means the processor does not loose time due to performance bottlenecks like the cache misses or the branch mispredictions. Each element of my C++ implementation of P is above 1.0 which is fine. The key point of this Figure is the linear growth in the number of RI for all the methods and with respect to the number of NPCs. It ensures that solution plans are not cached, that the processor fully applies the planner to solve all given problem instances.

With such performances, the use of the cache undeniable, however. In light of these last results, we can conclude that the cache is not used to store solution plans but that it may be used to cache the entire planner and all operators in the problem being evaluated to not loose time while executing the code, i.e. while planning.

Discussion

In this subsection, I discuss the creation of the benchmarks and the runtime performance difference between P and P.

• Although relaxing the syntactical restriction (S) with the structural restriction (C * 2 ) gives more freedom to the representation of operators, there are still concessions to be made. For instance, it is not possible to have the following situation: let consider two switches On/Off where the two On operators are required by an operator A and the two Off required by another operator B. All the operators are concerned by the restriction (C * 2 ) and the two switches must be removed one after another from the operator graphs with the red directed edges to see whether (C * 2 ) is respected. It is not respected because there is a path between A and B and a path between B and A in the resulting operator graph with the red directed edges (linked to threatening actions) after removing one switch. In such situation, the trick is to merge the two switches in one. This is what I did with the Native Hunter, with the operators Unequip and GetReadyToHunt. Initially, I provided the Native hunter with the operators EquipBow, UnequipBow, EquipQuiver, UnequipQuiver. To GoHunting the Native Hunter had to EquipBow and EquipQuiver and, then, to StoreTheGame the Native Hunter had to UnequipBow and UnequipQuiver. This model is equivalent to the two switches On/Off situation previously described and does not respect (C * 2 ). Therefore, EquipBow and EquipQuiver were merged into GetReadyToHunt and UnequipBow and Un-equip Quiver were merged into Unequip. (C * 2 ) is thus now respected. Eventually, I decided to create a SAS-PUSC 2 version by swapping the order between Store the Game and Unequip. Initially, the Native Hunter had to Unequip before StoringTheGame, in the final version (cf. Table F.7) this is the opposite.

• To collect the data presented in Figures 5.16, 5.17 and 5.18, I have set up experiments with a fixed start for each benchmark (a fixed goal for HZD), and with no more than 10 goals (or 6 starts for HZD). in other words, there is no more than 10 different problem instances per benchmark. Which implies for 3, 4 millions NPCs that problem instances were repeated at least 340 thousands times. I showed in subsection 5.2.4 that my AMD Ryzen™ 7 2700X does not cache the solution plans of the problem instances, however, but applies the planner completely every time despite the many repetitions. We did this internal analysis of the processor because the use of the function rand() from the C library to randomly choose a goal (ACO and RDR-2) or a start (HZD) is not relevant in these experiments due to the very short time interval Phas to work with. Even if the function rand() is called at each initialization of the start and the goal states, the time seed remains the same which implies the function rand() always return the same number, therefore rand() was not relevant. Then, the use of a for loop to choose one goal (or start for HZD) after another, or the use of a list of ten thousands unsorted number between 0 and the number of goals (or starts for HZD) to try to choose randomly a goal (or a start), produced eventually the same results.

• Data from P, shown in Figures 5.16 and 5.17 are a bit noisy compared to those of P, shown in Figure 5.18. The reason is that the two algorithms provide plans to a number of NPCs with a different order of magnitude. These data from P, although noisy, were not easy to obtain because of the very short time interval implied by the real-time constraint, however. Compare to the C++ implementation of P, I have not used the C++ Standard Library vector class to store operators, actions, plans and other useful internal lists for P. The reason is that the vector class has an internal container with a fixed-size that may allocate some extra storage to accommodate for possible growth [49]. Even though it is possible to allocate some extra storage by hand with the function reserve, it is not possible to reserve the size of the container in a vector of vector. When P was first coded with vectors, the dynamic allocation for the extra storage of the vectors were visible on the collected data. It created "jump" in the set of runtime data and the slope of each subset framed by two jumps was varying as well. In other words, the use of vectors with P made the collected data unusable. As a result, my implementation of P is coded with C functions like malloc or memset to create and allocate useful lists, variables, hashing tables and so on. Then, the internal functions (BuildChain, SearchExtension, TopoPlan, DFSTopo) use pointers and references as much as possible to avoid the manipulation of heavy objects, like actions. Besides, compared to P, actions are not instantiated while planning in my C++ implementation of P. Indeed, P simply uses references to actions/operators from the hashing table that was built in the setup. On the contrary, P plans for problems of the class SAS + -PUS and, for some problem instances, the minimal solution plans have two times the same actions in it. As a result, the planning algorithm P instantiates actions when building chain of actions and manipulates them during the entire planning process. In my C++ implementation of P I tried to manipulate actions as little as possible and use references instead. P returns a partial-ordered action plan A, R , with A the set of actions and R the set of orders between actions. Instantiations of actions in my C++ implementation of P occur only when building chains of actions, in order to insert them into A, which I represent as a vector of actions. Finally, R is a vector of integer vectors and it stores the orders between the references of the actions in A. Eventually, although I am convinced the way I have implemented P played a role in its performances, the quality of the AMD Ryzen™ 7 2700X processor played a role as well [50]. In particular, the AMD Ryzen™ 7 2700X has the Sense MI Technology which means this processor uses the Neural Net Prediction, a neural network whose purpose is to understand the applications and anticipate the next steps in the activity flow in realtime. This technology combined with many others, like pipelining, the use of dispatchers, the use of threads, etc. may have played a role as well.

• It seems that many factors explain the huge differences in performance between P and P. Obviously, the complexity reduction played a role, especially when m and n are high. However, when m is low (< 100), the difference in performance is huge and seems to grow, i.e. the lower m, the higher the difference in performance. The scalable problems highlight this point. P is 10 times faster than P in MultiPrv_n_Cycle when m is low, and about 2 times faster when m is high. In OnePrv_n_Cycle, P is 65 times faster than P when m is low, and 178 times faster when m is high. Even though these are significant differences in the runtime performance between the two algorithms, they are not as significant as the one on the realistic benchmarks (P is > 34.000 times faster than P). I performed an "AMD experiment", i.e. I checked the behavior of my processor when solving the realistic benchmarks, to ensure that plans were not cached. It was not the case, so they may be other explanations explaining the huge difference in performance. My intuition is that, when m is small, the problem is small enough for my AMD processor to cache all the P functions and the problem inside the L1 cache.

• Most of the time, there are several different topological sorts for a given partialorder. Adding some randomization while selecting a node (action) to sort can bring further diversity for solution action plans. For ACOrigins, if the Citizen must feed himself, he can: Buy Food, Eat, Buy Drink, Drink. But he also can: Buy Drink, Buy Food, Drink and Eat. And if one wants to reduce the possibilities, then additional prevail-conditions must be considered.

• Concerning the knowledge representation, there are some actions that must be executed before any other action. For instance with how I represent the Acquisition Machines of Horizon Zero Dawn: as they only have one goal: Harvest, if they are threatened, the first action that must be executed is obviously Attack(?target); therefore, (isThreatened = f alse) is a prevail-condition for any other action in my representation. We can do that with as many actions as we want, it will ensure that, no matter the topological sort in P, the result is a valid and totally ordered plan. The problem may become very constrained, however, as shown in Figure 5.14. So, I doubt this representation is easily sustainable for large projects, although it allows planners to create well-ordered action plans with more than 10 actions. The alternative is to rely on the Goal system to help to handle the behavior consistency. This is the case in F.E.A.R., you definitely do not want your NPCs to patrol when the player is spotted, so the Attack action must be executed before the Patrol action. The state (TargetIsDead = true) is not a precondition of Patrol, however. The Goal system handles the situation instead, by giving the goal KillEnemy, then by giving the goal Patrol when the scene is safe again.

• Given any SAS problem (Φ = (M, S, O)), the start s 0 and the goal s states create instances of the problem ( = (Φ, s 0 , s )). s 0 and s are a collection of defined state variables (SAS-structure), these states variables are said to model the world of the NPCs but they can be used for the sole purpose of planning. Let consider the Horse Breeder problem from Table F.5, the state variables Haystack and W ater and their respective domain are useful to create plans in order to fill the feeder and the water trough, respectively. But, to decide whether the feeder or the water trough need to be filled, other state variables, like F eederF illed and T roughF illed, can be considered. These are sensing state variables that the Horse Breeder may use to decide whether he needs to s (Feed the horses), or just s (Fill the feeder), or just s (Fill the trough). Furthermore, in subsection 5.2.1, I mentioned that Haystack and W ater can be seen as targetable items, so that the Horse Breeder can have item variables like targetHaystack : ?haystackItem or targetW ater :? sourceOf W ater that store positions of the last seen items or the closest ones if the Horse Breeder knows his surroundings for instance. This discussion allows to highlight some differences between a decision-making system and a planning system. An NPC may be equipped with many different variables: state ones, item ones, sensing ones, etc. Some of these variables may be used by the decision-making system while the planning system may use others. Some variables can obviously be shared by both systems. According to the belief, the needs, the will, etc. of an NPC, the decision-making system will define a goal s and a start s 0 for the planning system to return an action plan. Concerning targetable items, either the NPC is equipped with item variables that keep track of the useful items that surround him or, while executing the plan, the NPC seeks the required items. In any case, the search for an item, as well as the search for a path, must not be done while planning. This last point is about the temporal order of magnitudes. The planner works in real-time, a plan can be returned in few nanoseconds according to the results of section 5.2 whereas the execution time of an action is at least as long as its animation, i.e. several milliseconds or seconds. As a consequence, the planning system must not be burdened by any variable updates or searches. An NPC must either anticipate by updating as often as possible its variables or find what it needs while executing its plan. The second method may be less safe because it may lead to plan interruptions if requirements are not met while executing the plan. Whereas the first method can influence the decision-making system. For the Horse Breeder, let consider the following situation: the feeder is empty as well as his hay stock and he needs to wait until tomorrow for the supplier to restock, then targetHaystack =?none and F eederF illed = f alse. In this case, the Horse Breeder needs to s (Fill the feeder) but he cannot because he does not have haystacks anymore. Hence, the decision-making system will not give the goal s (Fill the feeder) to the Horse Breeder. On the contrary, with the second method, the decision-making system will give the goal s (Fill the feeder). Then, the execution of Take Haystack actions is equivalent to a search for an haystack, which will fail in this situation, thus forcing the Horse Breeder to interrupt the on-going plan. Goal-Oriented Action Planning I conducted a study on the GOAP planner of F.E.A.R. and found that action costs, which were used by the cost function to evaluate the consistence of a plan, are in practice overshadowed by context preconditions. So much so that the actions become contextually post-unique (1). Then, I also highlight from the source code of F.E.A.R. that its developers tried to keep the action representation simple, that is each action only has few preconditions and only one post-condition. Which means actions are unary (2) in F.E.A.R. Finally, I highlighted that action plans are totally ordered (3) and there is no two times the same action in them (4). I have finally turned these four points into assumptions that I have used throughout my thesis.

Simplified Action Structure I recall the theory behind the Simplified Action Structure (SAS) formalism in this chapter. Even though GOAP is based on STRIPS, it is in fact closer to the SAS formalism. Furthermore, ( 1) and ( 2), two of my assumptions, are restrictions that C. Bäckström have used to create SAS + -PUS problems, which are problems that can be solved by a polynomial time algorithm. As I aimed to reduce the complexity for scaling up realtime planning, I therefore implemented the SAS + -PUS algorithm in C++, denoted P, and I tried it on realistic benchmarks, based on video games, which form SAS + -PUS problems. The results were encouraging as P is able to plan for hundreds of NPCs in realtime and by providing them longer action plans that current planners in GOAP system can do. The restriction (S), however, is very restrictive.

Topological Planning In this chapter, I relaxed (S) and presented new structural restrictions to be combined with (P) and (U) so as to create tractable problems for video games. As one of my assumption is no two times the same action in a solution plan, I introduced an output restriction denoted Υ 1 and which I use to create the Υ 1 -Truth Crite-rion, a truth criterion which, if respected, ensures the solution plan is correct. Υ 1 does not provide instructions to produce tractable problems, however. I therefore introduce a new restriction, denoted C k , whose role is to restrict to at most k the number of requestable operators inside the unique cycle of each domain operator graph. Indeed, I demonstrated in this chapter that the reason of the intractability of SAS-PU problems is operators affecting the same state variable and looping together. I therefore closely study and restrict, with C k , the cycles formed by operators looping together. In particular, I present the classes of problems SAS-PUC 0 , SAS-PUC S 2 and SAS-PUC * 2 , and I proved that these problems are tractable because their solvable instances are solved by minimal solution plans that are Υ 1 , i.e. they do not have twice the same action in them. Eventually, I present TopoPlan, denoted P, a linear time algorithm that solves instances of a SAS-PUC 0 , SAS-PUC S 2 or SAS-PUC * 2 problem. I proved that this algorithm is correct with respect to the Υ 1 -Truth Criterion, and complete. I also proved that P as a worst-case space complexity of O(n 2 • m 2 ), i.e. quadratic in the number of state variables times the size of the variable domain. Finally, I proved that P's worst-case time complexity is O(|O| + |E O |), i.e. linear in function of the number of operators plus the number of orders between the operators. I therefore succeeded in answering one part of my main question, the one on how to reduce time complexity.

Benchmarks and Results

In these chapter, I present the runtime performance of both P and P on different benchmarks. I first test P and P on scalable problems, i.e. problems whose resolution difficulty grows in function of the number of state variables or in function of the size of each state variable domain. These benchmarks allow me to show empirically the time complexity of both P and P. It also shows that, overall, P is faster than P.

Then, I introduced realistic benchmarks: RDR-2, ACO and HZD, which are based on the commercial video games Red Dead Redemption, Assassin's Creed: Origins and Horizon Zero Dawn respectively. These benchmarks allow me to show that my tractable classes of problems are applicable to realistic problems. In particular, they allow the creation of problems that were not possible with SAS + -PUS and that are closer to real-world problems. The runtime performance of P on some instances of these realistic problems were astounding, P was able to produce millions of plans in real-time. With adapted instances that can be solved by both P and P, I show that P is way faster than P to solve them.

The creation of operators remains tricky with my classes of problems, however, as some concessions may be required to stay within the restrictions. It is worth the effort, however, because instances of such problems are solved by P, my linear-time algorithm with which I have proven that once optimized it is possible to plan in real-time for millions of NPCs. Eventually, the main conclusion in light of the performances of P is that planning in realtime on CPU is feasible on a large scale with current technologies.

Future Work

In this section, I give some ideas to pursue the researches presented in my PhD manuscript. SAS-PUC k problems that are Υ 2 The next step after SAS-PUC k problems that are Υ 1 are obviously those that are Υ 2 , then Υ 3 and so on. In my opinion, the study of the unique cycle of the domain operator graph remains crucial for those who want to explore these problems.

Another step is to consider SAS * -or SAS + -structures with the restrictions (P), (U) and C k . These two structures allow state variables to be undefined in the goal state, while the SAS + also allows start state to be undefined. It adds a notion of uncertainty which can be of interest for some people.

Hierarchical Topological Planning

In the first paragraph of subsection 5.2.5, concerning the Native Hunter, I discussed how I have merged the pair of operators Equip-Bow/UnequipBow and EquipQuiver/UnequipQuiver into the pair GetReadyToHunt/Unequip because the two first pairs of operators did not respect the restriction C * 2 . But, GetReady-ToHung and Unequip can in fact be seen as sub-goals whose minimal solution plans are EquipBow, EquipQuiver and UnequipBow, UnequipQuiver respectively. This observation paves the way to hierarchical planning with P. One can creates problems of sub-problems of sub-sub-problems... and then applies P to solve each problem instance after another in order to get a minimal solution plan for a given agent.

Apply TopoPlan in GOAP systems

It is a matter of maturity in order to increase the Technology Readiness level of P. I have proved theoretically and empirically my algorithm and I have applied it on realistic benchmarks and it has astonishing result. Integrating it into a project while ensuring that it works and interacts well with other subsystems is certainly a next step.

Procedure 5 FindPost(H V , x, i) Input:H V , vector implementation of H; x ∈ D v i . Parameters: a, action. Output:a, instantiated action from H.

1: a ← Instantiate(H V [i][x]) 2: return a Procedure 6 BuildChain(H V , x, g, v i , A, T, R, U V )
Input: H V , A, T, R, U V are called by reference; x, g ∈ D v i . Parameters: C, chain of actions; a, an action. Output: C. 2 restriction: the ordering of a green (resp. blue) action after a yellow one is not dictated by a blue (resp. green) action. That is, the planner can order a green (resp. blue) action right after its yellow establisher and/or promote it right before its yellow clobberer. If a blue operator is related to a green one, however, then the planner must respect the order in some problems, which can lead to cases whose minimal solution plan is not Υ 1 . Figures C.3 and C.4 present such structures having relations between blue and green operators without passing through the yellow parts. In such cases, the planner must clobber the prevail-condition of some blue (resp. green) actions to establish a green (resp. blue) one that must be ordered before, due to the relation in the operator graph. The planner must then establish previously clobbered prevail-conditions to complete the solution plan and so on, until the goal state is reached with a correct and minimal plan. It results in minimal solution plans that are not Υ 1 , such as C.3c and C.4c. The problems differ with s 0 [v i ] and s [v i ] defined in the sub-caption. Finally, we have: In this problem, a 2 must be demoted after a + because a + establishes a 1 which itself establishes a 2 . Then, v i = -is not the goal state, so a 2 is promoted before a + ' which satisfies the pre-condition of

1: if x g then return 2: end if 3: a ← F indP ost(H V , g, v i ) 4: if a = ∅ or U V [g] then Fail. 5: end if 6: U V [g] ← true 7: C ← BuildChain(H V , x, pre(a)[v i ], v i , A, T, R, U V ) 8: U V [g] ← f alse 9: A ← A ∪ {a}; T ← T ∪
v i ← Af f ects(h) 12: H V [v i ][post(h)[v i ]] ← h 13: end for 14: U V ← size max v i ∈M (D v i )
γv i ← BuildChain(H V , s 0 [v i ], s [v i ], v i , A, T, R, U V
a+ δv i (+,s*) (c) s 0 [v i ] = -; s [v i ] = post(δ v i (+, s [v i ])) a 1 a 2 a+ a- δ vi (+,s * ) ∨ a 1 a 2 a+ a- δ vi (+,s * ) (d) s 0 [v i ] = + ; s [v i ] = post(δ v i (+, s [v i ])) a 1 a 2 a- δ vi (-,s * ) (e) s 0 [v i ] = + ; s [v i ] = post(δ v i (-, s [v i ])) a 1 a 2 a+ a- δ vi (-,s * ) ∨ a 1 a 2 a+ a- δ vi (-,s * ) (f) s 0 [v i ] = -; s [v i ] = post(δ v i (-, s [v i ]))
pre(o + v i ) = post(o - v i ) = -; pre(o - v i ) = post(o + v i ) = + ; prv(o 1 )[v i ] = + and prv(o 2 )[v i ] = -.
δ vi (+,s * ) a+' (c) s 0 [v i ] = -; s [v i ] = post(δ v i (+, s [v i ])) a 1 a 2 a- δ vi (-,s * ) (d) s 0 [v i ] = + ; s [v i ] = post(δ v i (-, s [v i ])) a 1 a 2 a+ a- δ vi (+,s * ) (e) s 0 [v i ] = + ; s [v i ] = post(δ v i (+, s [v i ])) a 1 a 2 a+ a- δ vi (-,s * ) (f) s 0 [v i ] = -; s [v i ] = post(δ v i (-, s [v i ]))
δ v i (+, s [v i ]). Finally, δ v i (+, s [v i ]) satisfies s [v i ].
v k+1 ] = +) is δ v i (s 0 [v k+1 ], +) and δ v i (s 0 [v k+1 ], +) = because (v k+1 = +) cannot be reached from s 0 [v k+1 ] ∈ D v k+1 \ {-,
) ∩ Cycle(v 1 ) = Cycle(v 1 ) = {o - v 1 , o + v 1 }. Due to C *
2 , we therefore have:

• Let O(v 1 ) = (O(v 1 ) \ Cycle(v 1 )), (O(v 1 ) , E pre (O(v 1 )
)) is a domain operator graph with two distincts, possibly empty, directed trees (cf. subsection 4.2.4)

• ∀o 1 ∈ O \ O(v 1 ) s.t. o + v 1 ≺ prv o 1 , o 1 is part of G 1 = (O 1 , E + O 1 ) ⊂ G O\Cycle(v 1 ) which is a declobbered operator graph of a SAS-PUC 0 or SAS-PUC S 2 structure. • ∀o 2 ∈ O \ O(v 1 ) s.t. o - v 1 ≺ prv o 2 , o 2 is part of G 2 = (O 2 , E + O 2 ) ⊂ G O\Cycle(v 1 ) which is a declobbered operator graph of a SAS-PUC 0 or SAS-PUC S 2 structure. Indeed, due to C * 2 , o 1 is not related to o 2 in G O\Cycle(v 1 )
. Which means both operators are in two distinct subgraphs of G O\Cycle(v i ) , namely G 1 and G 2 . Only the cycle Cycle(v 1 ) has two requestable operators in it, which implies that, ∀v i ∈ M \ {v 1 }, Cycle(v i ) is either empty (SAS-PUC 0 ), or has no requestable operator (SAS-PUC 0 ), or only have one requestable operator in it (SAS-PUC S 2 ). Therefore, for all solvable problem instances (1) such that s 0 [v 1 ] / ∈ {-, +}, then (1) is a SAS-PUC 0 or a SAS-PUC S 2 problem instance which implies that the minimal solution plan is Υ 1 (Theorems 4.4.3 and C.1.1). I will now consider s

0 [v 1 ] ∈ {-, +} and s [v 1 ] ∈ {δ -, δ + } with δ -= post(δ v 1 (-, δ -)) and δ + = post(δ v 1 (+, δ -)) such that a + v 1 , a - v 1 / ∈ δ v 1 (-, δ -) and a + v 1 , a - v 1 / ∈ δ v 1 (+, δ + ), i.
e. no action of these two minimal chains are instantiated from an operator of Cycle(v 1 ) (cf. Appendices C.1a and C.3a to illustrate δ -and δ + ). These minimal chains are respectively empty if δ -= -and δ + = + respectively. These starts and goals correspond to every remaining combination of the pair (s 0 [v 1 ], s [v 1 ]). I denote α 1 the sequence of actions such that: ∃a ∈ α 1 , prv(a)[v 1 ] + ∧ ∀a ∈ α 1 , prv(a ) = +. I denote α 2 the sequence of actions such that: ∃b ∈ α 2 , prv(b)[v 1 ] -∧ ∀b ∈ α 2 , prv(b ) = +. α 1 and α 2 are necessarily Υ 1 as they are minimal solution plans of SAS-PUC S 2 or SAS-PUC 0 subproblem instances. Depending on the combination of (s 0 [v 1 ], s [v 1 ]), (1) can only have the following types of plan ∆:

• (s 0 [v 1 ], s [v 1 ]) = (-, δ -), the minimal solution plan is: ∆ = a + v i ⊕ α 1 ⊕ a - v i ⊕ α 2 , with: chain v 1 (-, δ -) = a + v i , a - v i ⊕ δ v 1 (-, δ -) ∧ δ v 1 (-, δ -) ⊆ α 2 . or, ∆ = α 1 ⊕ α 2 , with: chain v 1 (-, δ -) = δ v 1 (-, δ -) ⊆ ∆ ∧ ∀a ∈ ∆, prv(a)[v 1 ] = +. • (s 0 [v 1 ], s [v 1 ]) = (-, δ + ), the minimal solution plan is: ∆ = α 2 ⊕ a + v i ⊕ α 1 , with: chain v 1 (-, δ + ) = a + v i ⊕ δ v 1 (+, δ + ) ∧ δ v 1 (+, δ + ) ⊆ α 1 . • (s 0 [v 1 ], s [v 1 ]) = (+, δ -), the minimal solution plan is: ∆ = α 1 ⊕ a - v i ⊕ α 2 with: δ v 1 (-, δ -) ⊆ α 2 ∧ chain v 1 (-, δ + ) = a - v i ⊕ δ v 1 (-, δ -). • (s 0 [v 1 ], s [v 1 ]) = (+, δ + ), the minimal solution plan is: ∆ = ⊕ a - v i ⊕ α 2 ⊕ a + v i ⊕ α 1 with: chain v 1 (+, δ + ) = a - v i , a + v i ⊕ δ v 1 (+, δ + ) ∧ δ v 1 (+, δ + ) ⊆ α 1 . or, ∆ = α 2 ⊕ α 1 , with: chain v 1 (+, δ + ) = δ v 1 (+, δ + ) ⊆ ∆ ∧ ∀a ∈ ∆, prv(a)[v 1 ] = -.
Every problem instance (1) have therefore a minimal solution plan that is Υ 1 , which concludes the base case.

Induction hypothesis: Let k ∈ N such that k ≤ |M|. The proposition at rank k is: "Any (k) = (Φ (k) , s 0 , s ), a SAS-PUC * 2 problem instance with Φ (k) a SAS-structure that has exactly k cycles restricted by C * 2 , has a minimal solution plan that is Υ 1 ." Let consider {v 1 , ..., v k } ⊆ M the state variables such that, for each

v i ∈ {v 1 , ..., v k }, Cycle(v i ) is restricted by C * 2 , we have RO(v i ) ∩ Cycle(v i ) = Cycle(v i ) = {o - v i , o + v i }. Induction step: Let consider v k+1 , the (k + 1) th state variable restricted by C * 2 . We have: RO(v k+1 ) ∩ Cycle(v k+1 ) = Cycle(v k+1 ) = {o - v k+1 , o + v k+1 }. The restriction C * 2 implies that: • Let O(v k+1 ) = (O(v k+1 ) \ Cycle(v k+1 )), (O(v k+1 ) , E pre (O(v k+1 ) )
) is a domain operator graph with two distincts, possibly empty, directed trees (cf. subsection 4.2.4)

• ∀o 1 ∈ O \ O(v k+1 ) s.t. o + v k+1 ≺ prv o 1 , o 1 is part of G 1 = (O 1 , E + O 1 ) ⊂ G O\Cycle(v k+1
) which is a declobbered operator graph of a SAS-PUC 0 , SAS-PUC S 2 structure or it is a declobbered operator graph of the structure of a SAS-PUC * 2 problem with exactly k cycles restricted by C * 2 (Induction hypothesis).

•

∀o 2 ∈ O \ O(v 1 ) s.t. o - v 1 ≺ prv o 2 , o 2 is part of G 2 = (O 2 , E + O 2 ) ⊂ G O\Cycle(v i )
which is a declobbered operator graph of a SAS-PUC 0 , SAS-PUC S 2 structure or it is a declobbered operator graph of the structure of a SAS-PUC * 2 problem with exactly k cycles restricted by C * 2 (Induction hypothesis).

With an explanation similar to the base case, i.e. via the enumerated possible minimal solution plans with respect to all the combinations of (s 0

[v k+1 ], s [v k+1 ]), I prove that chain v k+1 (s 0 [v k+1 ], s [v k+1 ]) is always Υ 1 .

D

Proof of Theorems in Chapter 4

This appendix presents the proof of BuildChain and TopoPlan 4. In the following, O represents the set of (PU) actions stored in the hashing table that was built in the setup (cf. subsection 4.5.1). Each action of O has a color:

• (White) the action is not instantiated, i.e. it is not in A. Every action of O is white at the beginning of the resolution of any problem instance ;

• (Yellow) the action is instantiated, i.e. it is in A ;

• (Blue) the action is being topologically sorted ;

• (Green) the action is topologically sorted, i.e. added in a linear solution plan ∆. ∆ = (A, E A ) is a solution plan made up of: A a set of (PU) actions, E A a set of orders between the actions of A.

D.1 Proof of BuildChain

Definition D.1.1. The specification of BuildChain is:

• Input: O, A, E A , v i ∈ M, s, g ∈ D v i s.t. s = g and a g v i ∈ O is white.
• Output: a minimal chain of actions from s to g, δ v i (s, g). Each action a of δ v i (s, g) is colored yellow, added to A and each order (b, a), s.t. b ∈ N pre (a), is added to E A .

Theorem D.1.1. BuildChain satisfies its specification, so it is correct.

Before proving the theorem, I provide some prerequisites and explanations. Let v i ∈ M, s, g ∈ D v i , a chain of actions on v i from s to g is a linear sequence of actions such that:

• chain v i (s, g) = (D, E D ) = a |D| , ..., a 2 , a 1 • E D = {(a k+1 , a k ) | k ∈ [[1, |D| -1]], a k+1 ∈ N pre (a k )} = {(a |D| , a |D|-1 ), ..., (a 2 , a 1 )} • D ⊆ A(v i ) ⊆ A • E D ⊆ E A(v i ) ⊆ E A
The procedure BuildChain adds an additional order when building the chain of actions. Let denote E + D this set with the additional order. It is defined as follows:

E + D = {(a |D|+1 , a |D| ) | a |D|+1 ∈ N pre (a |D| )} ∪ E D It should
(1.4) A ← A ∪ {a g v i } ; E A ← E A ∪ {(a y v i , a g v i )} • (1.13) A ← A ∪ {a y v i } ; E A ← E A ∪ {(a y v i , a x v i )}
could have respectively been replaced by:

• (1.4) D ← D ∪ {a g v i } ; E + D ← E + D ∪ {(a y v i , a g v i )} • (1.13) D ← D ∪ {a y v i } ; E + D ← E + D ∪ {(a y v i , a x v i )}
• And after "(1.17): end while", in a new line, the instructions:

A ← A ∪ D; E A ← E A ∪ E + D , could have been added.
I will use this equivalence to prove that the procedure BuildChain 2 is correct.

Proof. of Theorem D.1.1 Loop Invariant: Let k ∈ N * , P(k) "at the end of the k th loop, chain v i (k, 0) = δ v i (k, 0), i.e. it is minimal such that:

• δ v i (k, 0) = (D (k) , E + D (k-1) ) = a k-1 v i , ..., a 0 v i ; • E + D (k) = {(a k v i , a k-1 v i )} ∪ E + D (k-1) and a k v i ∈ N pre (a k-1 v i ) ;
that situation, the Υ 1 -Truth Criterion allows two possibilities: an establishment order (a q v j ≺ a p v i ) or an promotion order (a p v i ≺ N ext(a q v j )). I decided to use exclusively the establisment order (3.28) and it was possible only because of the structural properties induced by C * 2 . Let q = + andthe other requestable value of Cycle(v j ), C * 2 insures that actions having q = + as a prevail-condition (like a p v i ) are in a different sequence of actions that the actions havingas a prevail-condition (cf. Proof 4.4.6). The two orders are therefore both correct.

In addition to that, if an establisher does not exist (3.12) or if BuildChain, (3.15) or (3.21), fails, then the problem instance is not solvable.

Eventually, each prevail-condition q of a p v i on v j is properly asserted. DropHaystack of the HorseBreeder problem for example). In that case, a p v i has no establisher via post-prv dependency and it is therefore obviously asserted with respect to the Υ 1 -Truth Criterion. Therefore the for loop (3.10) is correct and any action of A is correctly established and declobbered by (3.10) in function of the Υ 1 -Truth Criterion.

Due to Υ 1 and the post-uniqueness, the ordering of an action a p v i ∈ A in ∆ 2 is independent of other actions of A. And because (3.10) is correct, i.e. it correctly establishes and declobbers every defined prevail-condition of a p v i ∈ A, it results that every action of A is correctly ordered in ∆ 2 . Therefore, P (3.9) (l) holds for every l ∈ [ [1, |A|]]. Then, |A| ≤ |O| due to Υ 1 and the if condition (1.11) of BuildChain which ensures a chain of actions cannot be Υ 2 of higher (cf. Corollary D.1.2). Therefore, the for loop (3.9) always terminates if (3.10) does not fail. And if (3.10) fails, it means the problem instance was not solvable. Eventually, P (3.9) (|A|) holds, therefore the for loop (3.9) is correct with respect to the Υ 1 -Truth Criterion: each action a ∈ A is properly ordered in ∆ 2 .

Phase 3: Although ∆ 2 respects the Υ 1 -Truth Criterion, it is possible that the problem instance is not solvable due to actions looping between each other in ∆ 2 . That is, if ∆ 2 is not irreflexive, then the problem instance is not solvable. This verification is done during Phase 3 with the topological sort. If Phase 3 succeeds, then ∆ 2 is irreflexive. A topological sort is a well-known algorithm, and it has been proven correct. Therefore, Phase 3 is correct and if it does not fail, then the result is: ∆ a Υ 1 , linear, minimal solution plan that solve the problem instance.

Eventually, TopoPlan satisfies its specification and it is therefore correct and complete.

E

Step

by Step Examples of P

In this Appendix, I present step by step how P works to solve two differetns problem instances of the Horse Breeder (Φ HB , cf. Table 4.2).

• 1 = (Φ HB , s 0 = 0, 0, 0 , s = 2, 0, 2 )

• 2 = (Φ HB , s 0 = 1, 0, 0 , s = 2, 0 , 2 )
In both instances, the horse breeder has to fill both the trough and the horse feeder. The difference between 1 and 2 lies at the start: in 1 the horse breeder carries nothing while in 2 he already has a haystack in hands.

Resolution of 1 = (Φ HB , s 0 = 0, 0, 0 , s = 2, 0, 2 ) by P :

Phase 1:

• s 0 [v 0 ] = s [v 0 ], BuildChain builds backwards (3.4): -FillHorseFeeder satisfies s [v 0 ], it is colored yellow and added to A. (TakeHaystack, FillHorseFeeder) is added to E A . N ext(TakeHaystack) = FillHorseFeeder is defined. pre(FillHorseFeeder) = s 0 [v 0 ].
-TakeHaystack satisfies pre(FillHorseFeeder), it is colored yellow and added to A.

(DropHaystack, TakeHaystack) is added to

E A . N ext(DropHaystack) = TakeHaystack is defined. pre(TakeHaystack) = s 0 [v 0 ]. -BuildChain ends, δ v i (s 0 [v 0 ], s [v 0 ]) = TakeHaystack, FillHorseFeeder has been built. • s 0 [v 1 ] = s [v 1 ], δ v i (s 0 [v 0 ], s [v 0 ]) = . • s 0 [v 2 ] = s [v 2 ],
v 14 = 1 v 14 = 2 u, 1, 2, 1, u Take Position o 0 v 15 v 15 = 1 v 15 = 0 u, u, u, u, u Store Riffle o 1 v 15 v 15 = 0 v 15 = 1 u, u, u, u, u Equip Riffle o 1 v 16 v 16 = 0 v 16 = 1 u, u, u, u, u Get Ammo o 2 v 16 v 16 = 1 v 16 = 2 u, 1, u, u, u Reload Riffle o 0 v 17 v 17 = 1 v 17 = 0 u, u, u, u, u Drop Bandit Bag o 1 v 17 v 17 = 0 v 17 = 1 u, u, u, u, u Equip Bandit Bag o 1 v 18 v 18 = 0 v 18 = 1 2, 1,
v 28 = 1 v 28 = 0 u, u Go rest o 1 v 28 v 28 = 0 v 28 = 1 u, u Hang out o 1 v 29 v 29 = 0 v 29 = 1 1, u Get a Drink o 2 v 29 v 29 = 1 v 29 =
v 23 = v 23 = 1 u, u, u, u, u Have a Meal o 0 v 24 v 24 = v 24 = 0 u, u, u, u, u Use Bathroom o 1 v 25 v 25 = v 25 = 1 u, u, u, u, u Relax o 1 v 26 v 26 = v 26 = 1 u, u, u, u, 3 Getting Clean o 2 v 26 v 26 = v 26 = 2 1, 0, u, u, 3 Rest o 1 v 27 v 27 = v 27 = 1 u, u, u, u, u Prepare for Work o 2 v 27 v 27 = v 27 = 2 u, u, 1, u, u Introduce then Dance o 3 v 27 v 27 = v 27 = 3 u, u, u, u, u Get paid v 23 : hunger, D v 23 = {0 : hungry, 1 : f ed}. v 24 : needBathroom, D v 24 = {0 : f alse, 1 : true}.
Pre Post Prevail v 3 , v 4 , v 5 , v 6 , v 7 , v 8 , v 9 Explanation a 0 v v 3 = 1 v 3 = 0 u, u, u, u, 0, u, u Store Horse Cleaning Tools a 1 v v 3 = 0 v 3 = 1 u, u, u, u, 0, u, u Get Horse Cleaning Tools a 0 v v 4 = 1 v 4 = 0 u, u, u, u, u, u, u Untie the Horse a 1 v v 4 = 0 v 4 = 1 u, u, u, u, u, u, u Tie the Horse a 2 v v 5 = 0 v 5 = 1 1, 1, u, u, u, u, u Roughly Clean the Horse Foot a 3 v v 5 = 1 v 5 = 2 1, 1, u, u, u, u, u Take Horse Shoe Off a 4 v v 5 = 2 v 5 = 3 1, 1, u, u, u, u, u Clean the Barefoot a 0 v v 5 = 3 v 5 = 4 1, 1, u, u, u, u, u Put Horse Shoe On a 1 v v 6 = 0 v 6 = 1 u, u, u, u, 1, u, u Light the Forge a 0 v v 7 = 1 v 7 = 0 0, u, u, u, u, u, u Store Forge Tools a 1 v v 7 = 0 v 7 = 1 0, u, u, u, u, u, u Take Forge Tools a 1 v v 8 = 0 v 8 = 1 u, u, u, u, 1, u, u Put Old Shoes in the Forge a 2 v v 8 = 1 v 8 = 2 u, u, u, u, 1, u, u
Wait for Shoes to be Hot

a 1 v v 9 = 0 v 9 = 1 u, u, u, u, 1, u, u Shape Hot Shoes v 3 : CleaningT ools, D v 3 = {0 : stored, 1 : equiped}.
O Pre Post Prevail v 0 , v 1 , v 2 Explanation o 0 v 0 v 0 = 1 v 0 = 0 u, u, u Drop Haystack o 1 v 0 v 0 = 0 v 0 = 1 u, 0, u Take Haystack o 2 v 0 v 0 = 1 v 0 = 2 u, u, u Pick up Bucket o 0 v 1 v 1 = 1 v 1 = 0 u, u, u Drop Bucket o 1 v 1 v 1 = 0 v 1 = 1 0, u, u Pick up Bucket o 1 v 2 v 2 = 0 v 2 = 1 u, 1, u Fill Bucket with Water o 2 v 2 v 2 = 1 v 2 = 2 u, 1, u Fill Horse Trough v 0 : HayStack, D v 0 = {0 : none, 1 : inHands, 2 : inF eeder}. v 1 : Bucket, D v 1 = {0 : none, 1 : inHands}. v 2 : W ater, D v 2 = {0 : inSource, 1 : inBucket, 2 : inT rough}.
v 19 = 0 v 19 = 1 u, u, u, u Roll a Barrel o 1 v 20 v 20 = 0 v 20 = 1 u, u, u, u Wash the Glass o 2 v 20 v 20 = 1 v 20 = 2 1, u, u, 1 Serve a Drink o 1 v 21 v 21 = 0 v 21 = 1 u, u, u, u Clean Room o 2 v 21 v 21 = 1 v 21 = 2 u, u, u, 1 Assign Room o 1 v 22 v 22 = 0 v 22 = 1 u, u, u, u Collect Cash v : DrinksF ountain, D v 19 = {0 : empty, 1 : f illed}. v : Glass, D v 20 = {0 : dirty, 1 : washed, 2 : served}. v : Room, D v 21 = {0 : dirty, 1 : cleaned, 2 : assigned}. v : M oneyCollected, D v 22 = {0 : f alse, 1 : true}.
v 10 = 0 v 10 = 1 u, 1, u, u Go Hunting o 2 v 10 v 10 = 1 v 10 = 2 u, u, u, u Store the Game o 0 v 11 v 11 = 1 v 11 = 0 2, u, u, u Un-equip o 1 v 11 v 11 = 0 v 11 = 1 u, u, 1, 1 Get Ready to Hunt o 0 v 12 v 12 = 0 v 12 = 1 u, u, u, u Rest o 0 v 13 v 13 = 0 v 13 = 1 u, u, u, u Drink (v 10 )

G

Résumé en Français

Cette thèse porte sur l'intelligence artificielle (IA) et la planification d'actions. En particulier, mes recherches portent sur la planification d'actions en temps réel et j'utilise les jeux vidéo pour tester, expérimenter et analyser mes résultats. L'IA est un domaine de connaissance et de recherche visant à concevoir des agents informatiques intelligents, c'està-dire des agents qui agissent intelligemment dans leur environnement et en fonction de leurs objectifs. À mon avis, la meilleure définition de l'IA, ou intelligence informatique, est celle de D. Poole, A. Mackworth et R. Goebel [1, p.1], traduite ici en Français : L'intelligence informatique est l'étude de la conception d'agents intelligents. Un agent est une entité qui agit dans un environnement -il fait quelque chose. Les agents comprennent les vers, les chiens, les thermostats, les avions, les humains, les organisations et la société. Un agent intelligent est un système qui agit de manière intelligente : Ce qu'il fait est approprié aux circonstances et à son objectif, il est flexible face à des environnements et des objectifs changeants, il apprend par l'expérience et il fait des choix appropriés compte tenu des limitations perceptives et des capacités de calcul limitées. L'objectif scientifique central de l'intelligence computationnelle est de comprendre les principes qui rendent possible un comportement intelligent, dans des systèmes naturels ou artificiels. L'hypothèse principale est que le raisonnement est un calcul. L'objectif central de l'ingénierie est de spécifier des méthodes pour la conception d'artefacts utiles et intelligents.

Cette introduction est organisée comme suit : je donne d'abord une définition de la planification en temps réel, je donne ensuite ma motivation suivie de mon approche pour répondre à la question principale : comment réduire la complexité temporelle pour augmenter la planification en temps réel ? Je termine l'introduction par le plan de ma thèse. La planification en temps réel concerne les systèmes de planification qui planifient dans un laps de temps très court, à savoir en une ou deux millisecondes tout au plus. Un bon exemple d'agents intelligents utilisant de tels planificateurs est celui des personnages non jouables (PNJ) qui peuplent certains jeux vidéo [2][3][4][5]. Pour ces jeux, le temps réel correspond à 10% du temps entre deux images, soit 3, 33 ms pour un jeu vidéo avec un taux de rafraîchissement de 30 images par seconde (FPS), 1, 67 ms pour un jeu à 60 FPS, 0, 83 ms pour un jeu à 120 FPS, et ainsi de suite. Comme je m'intéresse aux PNJ contrôlés par des planificateurs, je présente le système Goal-Oriented Action Planning (GOAP) car il s'agit de la technologie utilisée dans les jeux vidéo actuels pour planifier en temps réel (cf. Chapitre 2 du manuscrit). D'après une étude que j'ai menée sur GOAP, j'ai fait ressortir des usages qui peuvent être utilisés pour restreindre les problèmes de planification. Ces usages m'ont conduit à étudier le formalisme Simplified Action Structure [6] que je présente dans le Chapitre 3. J'utilise ensuite ce formalisme dans le Chapitre 4 et le Chapitre 5 comme base de travail.

Motivation :

Je crois fermement que la planification est un domaine de l'IA qui représente l'avenir du comportement des PNJ. Les jeux vidéo sont de plus en plus réalistes et beaux grâce aux graphismes. Les environnements de jeu passent des couloirs ( [9]. La contrainte temps réel signifie que le planificateur doit fournir autant de plans d'action que possible dans le temps imparti. Dans un jeu, le temps imparti est le temps entre deux images. Plus il y a d'agents à contrôler, moins il y a de temps pour générer un plan. De même pour le nombre d'actions dans un plan, plus il y a d'actions, plus il faut de temps pour générer un plan. La complexité temporelle est donc un critère important pour évaluer l'algorithme de planification. Malheureusement, le planificateur actuel utilisé dans GOAP est basé sur A*, un algorithme de recherche, et dont la complexité temporelle est exponentielle [7]. Cela conduit à des restrictions de développement pour respecter le budget temps tout en essayant de répondre aux attentes croissantes des joueurs. Ma question principale tout au long de cette thèse est donc : comment réduire la complexité temporelle pour permettre un passage à l'échelle de la planification en temps réel ?

Approche :

Pour répondre à cette question, j'ai d'abord étudié GOAP dans un jeu vidéo commercial. F.E.A.R., qui reste une grande référence en termes de PNJ crédibles, est également l'un des rares jeux vidéo commerciaux utilisant GOAP et dont le code source est librement disponible en ligne [10]. Sur la base des résultats de mon étude sur la planification dans F.E.A.R., j'ai ensuite analysé plusieurs dizaines de projets GitHub pour voir si GOAP est toujours mis en oeuvre de la même manière. Enfin, j'ai contacté certaines équipes de développeurs utilisant GOAP dans leurs jeux vidéo pour leur poser quelques questions sur leurs habitudes de développement. Tout cela m'a permis de tirer des conclusions que j'ai ensuite transformées en hypothèses.

Ces hypothèses m'ont ensuite conduit à étudier le formalisme SAS (Simplified Action Structure). En effet, deux de mes hypothèses sont en fait des restrictions d'entrée existantes du formalisme SAS. C. Bäckström, le fondateur de SAS, a restreint les problèmes de planification SAS pour trouver des sous-classes de problèmes abordables. Il a également développé un algorithme en temps polynomial qui résout les instances de ses problèmes abordables. J'ai réutilisé l'approche de C. Bäckström pour créer de nouvelles classes de problèmes adaptés aux PNJ dans les jeux vidéo et j'ai ensuite développé un algorithme potentiels jeux vidéo commerciaux en python, ruby et javascript. C++ et C# partagent 54 implémentations, qui ne se retrouveront pas toutes dans un jeu commercial, bien sûr ; mais si c'est le cas, alors ces planificateurs seront confrontés à ce qui est maintenant l'étape cruciale de l'utilisation de GOAP dans un jeu vidéo : la conception des coûts d'action du jeu pour lequel ils doivent générer des plans. En effet, l'utilisation de GOAP pour un nouveau jeu implique de nouvelles actions, et par conséquent de nouveaux coûts d'action à concevoir.

Le créateur de GOAP, Jeff Orkin, propose "d'appliquer un coût aux actions pour forcer A* à considérer des actions plus spécifiques avant des actions plus générales". Plus l'action est élevée dans la hiérarchie d'héritage C++, plus le coût de l'action est important ; et c'est effectivement ce que j'observe en étudiant le code source de F.E.A.R.

Conclusion :

J'ai analysé les données de planification collectées en jouant à F.E.A.R. pour évaluer l'utilisation des coûts d'action dans GOAP afin de fournir une réponse à l'important problème de développement de l'IA du jeu, à savoir la conception de ces coûts d'action au début d'une nouvelle partie. J'ai découvert que les comportements des PNJs sont le résultat des préconditions contextuelles : les coûts d'action ont un impact négligeable sur les comportements des PNJs. Par conséquent, si votre implémentation de GOAP suit de près celle de F.E.A.R., les coûts d'action peuvent être supprimés de cette implémentation en faveur des préconditions contextuelles ; cela se fait au prix de quelques cycles CPU, cependant, et le temps d'exécution des préconditions contextuelles ne doit pas être surutilisé mais surveillé de près pour satisfaire le budget temps de GOAP.

J'ai également analysé la manière dont l'heuristique de l'algorithme de recherche A* a été conçue et utilisée dans GOAP : J'ai expliqué qu'elle favorise en fait les plans d'action courts au lieu des plans d'action les moins chers, c'est-à-dire que le coût et la fonction heuristique n'évaluent pas la même métrique. Les jeux vidéo sont confrontés aux inconvénients majeurs de la planification, soulignés par D. Chapman [23], c'est non-abordable en général. Les industries du jeu essaient de contenir cette non-abordabilité en limitant le nombre d'appels par PNJ et par image [3,5,24] et en restreignant la taille des plans solution [21]. Malgré ces restrictions, les planificateurs dans GOAP restent des algorithmes de recherche comme A * [15] ou Djikstra [25] dont la complexité est exponentielle par rapport à la taille du plan de solution, ce qui en retour ne permet pas d'améliorer GOAP ou le système de planification en temps réel.

Finalement, même si les préconditions contextuelles dans F.E.A.R. diminuent les performances d'exécution du planificateur GOAP, je crois fermement que c'est la voie à suivre. En fait, parmi les réponses que nous avons obtenues en questionnant les équipes de développeurs avec la question : "Utilisez-vous des préconditions de contexte ?", il y a la réponse de Troy Humphrey (Highmoon Studio) qui nous a dit que ce genre de préconditions est mauvais parce que les développeurs vont mettre trop d'instructions à l'intérieur, y compris des fonctions lourdes comme le pathfinding (C'est le cas pour certaines actions de F.E.A.R.), même si on leur dit de ne pas le faire. Le résultat est que ces instructions peuvent être exécutées plusieurs fois pendant la planification car il n'y a pas de variable dédiée pour elles. C'est la deuxième partie de la réponse de Troy Humphrey, la forme actuelle des préconditions contextuelles n'est pas du tout optimale, elles doivent être stockées d'une manière ou d'une autre car la détection et la planification sont deux abstractions différentes. Finalement, en considérant qu'il existe une manière plus efficace de gérer les préconditions contextuelles et le fait qu'elles éclipsent les coûts d'action en pratique, je suis convaincu que les coûts d'action peuvent être entièrement supprimés en faveur des préconditions contextuelles. La conséquence directe sera que les actions deviendront contextuellement post-uniques.

J'ai donc fait les hypothèses suivantes que j'utilise dans cette thèse pour produire des classes de problèmes abordables et fournir un algorithme de complexité temporelle linéaire, correct et complet pour résoudre toute instance de ces classes.

Proposition G.2.1 (Working Assumptions). Les hypothèses suivantes sont compatibles avec la création d'un comportement réaliste pour les PNJ :

1. Les actions sont unaires.

Les actions sont contextuellement post-unique.

3. Les plans solutions sont totalement ordonnés.

Il n'y a pas deux fois la même action dans le plan.

La dernière hypothèse peut être surprenante à première vue. En fait, la taille des plans d'action est plutôt petite, par exemple 1 ou 2 actions au maximum dans F.E.A.R. Et elle est légèrement supérieure dans certains autres jeux vidéo utilisant GOAP. Et dans ces petits plans, en particulier dans F.E.A.R., il n'y a pas deux fois la même action.

G.3 Simplified Action Structure

Introduction : J'ai étudié le travail de C. Bäckström [6] pour répondre à ma question principale et pour créer un planificateur qui peut planifier en temps réel pour plus de quelques dizaines de PNJs et avec des plans plus longs. Je réutilise l'approche de C. Bäckström pour réduire la complexité temporelle. Il a restreint l'entrée des problèmes de planification afin d'en créer des abordables qui peuvent être résolus par un algorithme en temps polynomial. Avant d'aller plus loin dans sa direction, je rappelle la théorie derrière SAS et l'adapte à ma notation et à mes besoins. Je présente les classes de problèmes SAS-PU et SAS + -PUS. Ce dernier est une classe de problèmes traitables qui peuvent être résolus par un algorithme existant en temps polynomial. Je l'ai implémenté en C++ et testé sur des benchmarks réalistes.

La Simplified Action Structure (SAS), introduite par C. Bäckström [6] au début des années 90, est une version restreinte du formalisme Action Structure [26]. SAS est similaire au très connu formalisme propositionnel : Stanford Research Institute Problem Solver (STRIPS) [27], les opérateurs ont des post-conditions (post) mais : (i) les variables d'état sont multivaluées et (ii) les opérateurs ont deux types de préconditions. (i) Le caractère multivalué est une caractéristique du système de planification GOAP (cf. Chapitre 2). C'est en effet plus pratique que les listes d'ajout et de suppression du formalisme STRIPS, surtout lorsqu'on utilise un langage de programmation comme C++ qui permet l'affectation de valeurs. (ii) Les deux types de préconditions sont : les pré-conditions (pré) et les prevail-conditions (prv) (décrites dans ma thèse), et sont facilement adaptables à la représentation actuelle des opérateurs GOAP. La façon dont ces préconditions sont décrites est pratique d'un point de vue algorithmique car leur description permet une définition simple d'un Critère de vérité [23] qui est un critère utile pour prouver la correction et la complétude d'un algorithme. Finalement, les opérateurs de GOAP sont plus proches de ceux de SAS que de ceux de STRIPS. D. Chapman a prouvé que la planification est non-abordable en général [23] et, malheureusement, le système de planification GOAP ne fait pas exception (cf. Chapitre 2). L'approche de C. Bäckström pour faire face à l'intractabilité générale est d'appliquer des restrictions sur les opérateurs SAS afin de créer et d'étudier des sous-classes de problèmes dans l'espoir de trouver des problèmes utilisables et traitables [6,28]. Parmi ces restrictions figurent la (P)ost-uniqueness et la (U)nariness [29] qui concernent directement 2 de mes 4 hypothèses (cf. Proposition 2.4.1). Cependant, sans l'ajout d'autres restrictions, la classe de problèmes SAS-PU est NP-Hard [30]. L'ajout des restrictions (S)ingle-Valuedness et (B)inariness à (PU) conduit à SAS-PUBS et SAS + -PUS qui sont toutes deux des classes traitables et qui peuvent être résolues par les algorithmes en temps polynomial décrits dans la thèse de C. Bäckström [6].

Dans ce chapitre, je rappelle comment un problème de planification est décrit avec le formalisme SAS et ce qu'est un plan solution minimal pour une instance de problème SAS. Ensuite, je rappelle les restrictions existantes avant de me concentrer sur SAS-PU et SAS-PUS, deux classes de problèmes cohérentes pour GOAP selon mes hypothèses.

Conclusion :

La restriction (S) est très restrictive [35]. Il est délicat, voir difficile, de représenter des problèmes réalistes avec elle. (S) ne permet pas les problèmes avec deux actions dont l'une nécessite qu'une pièce allumée et l'autre que la pièce soit éteinte, par exemple. Cette restriction ne permet uniquement de dire que la pièce est allumée (resp. éteinte) en tant que prevail-condition. Supprimer (S) de SAS-PUS revient à considérer la classe de problèmes SAS-PU, qui est non-abordable. Le caractère non-abordable est liée aux plans solution minimaux de taille exponentielle. Il existe des problèmes où le plan solution minimal requiert un nombre exponentiel d'instances d'opérateurs, alors qu'avec (S), les plans solution minimaux ont au plus deux fois la même instance. Une de mes hypothèses du chapitre 2 est que les plans d'action dans les jeux vidéo ont au plus une occurrence de la même action, c'est-à-dire au plus une instance par opérateur. Est-il possible de ne considérer que les problèmes SAS-PU dont les plans solution minimaux ont au plus une instance par opérateur ? Existe-t-il une restriction aux structures SAS dont la classe résultante de problèmes avec (P) et (U) n'a que des plans solution minimaux avec une occurrence de la même action ?

G.4 La Planification Topologique Introduction

Ce chapitre est le coeur de cette thèse où je présente de nouvelles classes de problèmes et dont les instances sont résolues par mon algorithme, nommé TopoPlan et noté P, en temps linéaire. Dans ce chapitre, je présente d'abord les prérequis qui tirent avantage des restrictions (P) et (U). En particulier, je présente comment identifier les opérateurs et les actions en temps constant pendant la planification et comment les identifiants peuvent être utilisés pour créer de nouveaux ensembles de prédécesseurs. Ensuite, je présente les graphes d'opérateurs et les graphes d'actions qui sont des représentations graphiques de l'entrée et de la sortie d'un problème de planification SAS-PU. J'utilise ensuite ces graphes pour prouver le Υ 1 -Truth Criterion, un critère de vérité qui prend en compte mes hypothèses. J'utilise également ces graphes et certaines de leurs propriétés pour introduire une nouvelle restriction structurelle, notée C k , me permettant de créer de nouvelles classes de problèmes qui respectent mes hypothèses de travail et qui sont abordables. Je présente enfin mon algorithme en temps linéaire qui tire parti des prérequis et que je prouve correct et complet avec Υ 1 -Truth Criterion.

Dans le chapitre 2, j'ai défini quatre hypothèses (Proposition 2.4.1) qui m'ont aidé à atteindre la planification en temps réel. Parmi elles figurent les restrictions (P) et (U) qui, une fois combinées avec la structure SAS, créent la classe de problèmes SAS-PU, prouvée NP-Hard [30]. Dans le chapitre 3, je rappelle la restriction (S) qui, avec (P) et (U), créent la classe abordables des problèmes SAS-PUS. (S) est cependant très restrictive et certaines instances des problèmes SAS-PUS ont des plans solution minimaux avec deux instances d'une même action (Υ 2 ). J'étudie donc dans ma thèse la représentation SAS-PU et sa structure pour trouver des sous-classes SAS-PU qui sont abordables et Υ 1 , où Υ 1 signifie qu'il n'y a pas deux fois la même action dans le plan solution minimal de chaque instance résoluble d'un problème.

La propriété non-abordable de la classe SAS-PU est due à des plans solution minimaux de taille exponentielle. L'observation suivante de C. Bäckström [6, p. 175], traduite cidessous en Français, l'explique partiellement : L'une des sources de la planification non-abordable est que l'on autorise des instances qui nécessitent des solutions minimales de taille exponentielle. [...] Ce n'est cependant pas trivial (de supprimer de telles instances). Il n'existe pas d'ensemble unique de restrictions ayant cet effet. [...] Bien qu'il ne soit pas suffisant de limiter les plans pour atteindre le caractère abordable, c'est une étape inévitable à considérer.

Ce qui manque dans cette observation est la raison pour laquelle certaines solutions minimales sont des plans d'action de taille exponentielle. Une action est une instance d'un opérateur et, par conséquent, une solution minimale de taille exponentielle est un plan d'action où certains opérateurs ont été instanciés un nombre exponentiel de fois. Proof 6.14 [6, p. 138] en est un excellent exemple. La preuve est basée sur le Code Gray [37] représenté comme un problème SAS-PUB où chaque opérateur modifie un seul bit. Soit Φ = (M, S, O) une structure du problème Code Gray, l'instance = (Φ, s 0 = 0, 0, ..., 0 , s = 1, 0, ..., 0 ) conduit à un plan minimal de taille exponentielle principalement parce que l'opérateur qui fait passer le bit de poids faible de 0 à 1 est au moins instancié 2 (|M|-2) fois. Je montre un exemple à 3 bits du Code Gray dans mon manuscrit. C'est un exemple qui illustre bien comment un plan peut croître rapidement en taille. Les opérateurs du Code Gray peuvent être instanciés de nombreuses fois car les opérateurs affectant le même bit bouclent ensemble. • Le nombre d'actions du même opérateur dans ∆ est au plus k.

Prérequis

• Il y a au moins un opérateur instancié k fois dans ∆.

J'utilise Υ k pour définir les classes de problèmes SAS. Si un problème est Υ k , alors toutes les instances de ce problème ont un plan solution minimal au plus Υ k . Si une classe est Υ k , alors n'importe quel problème de cette classe est au plus Υ k . Par exemple, la classe SAS-PUS est Υ 2 et la classe SAS-PU est Υ ∞ . Bien évidemment, tout problème est au moins Υ 1 . Au vu de ma dernière hypothèse de travail, la question est : "Existe t'il des classes de problèmes au plus Υ 1 ?" Υ k est cependant une restriction de sortie. J'ai donc eu pour volonté de trouver une nouvelle restriction d'entrée qui, couplée à (P) et (U), permet de créer des classes de problèmes Υ 1 . Pour cela, j'ai décidé de modéliser les problèmes sous forme de graphe afin de les étudier et de trouver une nouvelle restriction structurelle.

Les Graphes d'Opérateur

De manière similaire aux state-transition graph et aux domain-transition graph [28], ou aux causal graphs [38,39], qui consistent à représenter un problème de planification sous forme de graphe, j'introduis trois nouveaux graphes qui se concentrent sur les opérateurs/actions et leurs relations entre eux : deux d'entre eux sont des graphes d'opérateurs

Critère de Vérité (Υ 1 -Truth Criterion)

Un plan solution minimal pour une instance de problème SAS-PU est composé de plusieurs chaînes d'actions entre l'état de départ et l'état but, et plus précisément il est composé d'une chaîne d'action (éventuellement vide) pour chaque variable d'état. Cette assertion de C. Bäckström tient lorsque (S) est relaxé, et vient de son critère de vérité dénommé Intended Minimal Plan (IMP). L'IMP, cependant, est adapté aux problèmes SAS + -PUS qui sont au plus Υ 2 . Dans cette section de mon manuscrit, je vise à fournir un critère de vérité pour les instances de problèmes SAS-PU dont les plans solution minimaux sont Υ 1 . La section dans mon manuscrit est organisée comme suit : je présente la forme générale d'une chaîne d'actions, c'est-à-dire les formes possibles d'une chaîne d'actions qui est Υ k . Sur cette base, je présente les formes possibles lorsque k = 1. Ainsi, une chaîne d'actions Υ 1 entre deux valeur x et z du domaine D v i , noté chain v i (x, z), a pour forme générique :

v i ∈ M, x, y, z ∈ D v i , chain v i (x, z) = δ v i (x, y) ⊕ δ v i (y, z) ∧ ∀a ∈ δ v i (x, y), ope(a) ∈ Cycle(v i ) ∧ ∀b ∈ δ v i (y, z), ope(b) / ∈ Cycle(v i )
Avec δ v i (x, y) la chaine minimale entre deux valeurs x et y, prouvé Υ 1 dans mon manuscrit. En fonction de divers paramètres, je présente l'ensemble des situations de clobbering (i.e. menace) à prendre en compte. Cela me permet de définir mon critère de vérité: Υ 1 -Truth Criterion, le critère à respecter pour ordonner un plan solution minimal d'une instance de problèmes SAS-PU qui serait Υ 1 . Dans la section présentant TopoPlan, mon algorithme de planification en temps linéaire, je me sers également de Υ 1 -Truth Criterion pour prouver la correction et la complétude de TopoPlan. • Pour collecter les données présentées dans le chapitre Benchmarks and Results, j'ai mis en place des expériences avec un départ fixe pour chaque benchmark (un but fixe pour HZD) et une dizaine de buts (une demi-douzaine de départs pour HZD). En d'autres termes, il n'y a pas plus de 10 instances de problème différentes par benchmark. Ce qui implique que pour 3, 4 millions de PNJs, les instances du problème ont été répétées au moins 340 milliers de fois. Je montre dans ce chapitre que mon procésseur, un AMD Ryzen™ 7 2700X, ne met pas en cache les plans solutions des instances du problème, cependant, mais applique le planificateur complètement à chaque fois malgré les nombreuses répétitions. J'ai effectué cette analyse interne du processeur parce que l'utilisation de la fonction rand() de la bibliothèque C pour choisir aléatoirement un but (ACO et RDR-2) ou un départ (HZD) n'est pas pertinente dans ces expériences en raison de l'intervalle de temps très court avec lequel P doit travailler. Même si la fonction rand() est appelée à chaque initialisation des états de départ et de but, la seed de temps reste la même. Ce qui implique que la fonction rand() renvoie toujours le même nombre, donc rand() n'est pas pertinente. Ensuite, l'utilisation d'une boucle for pour choisir un but (ou un départ pour HZD) après l'autre, ou l'utilisation d'une liste de dix mille nombres non triés entre 0 et le nombre de buts (ou de départs pour HZD) pour essayer de choisir aléatoirement un but (ou un départ), a finalement produit les mêmes résultats.

• Les données récoltées de P et présentées dans la thèse sont un peu bruitées par rapport à celles de P. La raison en est que les deux algorithmes fournissent des plans à un certain nombre de PNJ avec un ordre de grandeur différent. Ces données de P, bien que bruyantes, n'ont cependant pas été faciles à obtenir en raison de l'intervalle de temps très court impliqué par la contrainte de temps réel. Par rapport à l'implémentation C++ de P, je n'ai pas utilisé la classe vector de la bibliothèque standard C++ pour stocker les opérateurs, actions, plans et autres listes internes utiles pour P. La raison en est que la classe vector possède un conteneur interne de taille fixe qui peut allouer un espace supplémentaire pour faire face à une éventuelle croissance [49]. Même s'il est possible d'allouer un peu de stockage supplémentaire à la main avec la fonction reserve, il n'est pas possible de réserver la taille du conteneur dans un vecteur de vecteur. Lorsque P a été codé pour la première fois avec des vecteurs, l'allocation dynamique pour le stockage supplémentaire des vecteurs était visible sur les données collectées. Elle créait des "sauts" dans l'ensemble des données d'exécution et la pente de chaque sous-ensemble encadré par deux sauts était également variable. En d'autres termes, l'utilisation de vecteurs avec P rendait les données collectées inutilisables. Par conséquent, mon implémentation de P est codée avec des fonctions C comme malloc ou memset pour créer et allouer des listes utiles, des variables, des tables de hachage et ainsi de suite. Ensuite, les fonctions internes (BuildChain, SearchExtension, TopoPlan, DFSTopo) utilisent autant que possible des pointeurs et des références pour éviter la manipulation d'objets lourds, comme les actions. D'ailleurs, par rapport à P, les actions ne sont pas instanciées pendant la planification dans mon implémentation C++ de P Ėn effet, P utilise simplement les références aux actions/opérateurs de la table de hachage qui a été construite dans la configuration. A l'inverse, P planifie pour des problèmes de la classe SAS + -PUS et, pour certaines instances de problèmes, les plans solution minimaux contiennent deux fois les mêmes actions. Par conséquent, l'algorithme de planification P instancie les actions lors de la construction de la chaîne d'actions et les manipule tout au long du processus de planification. Dans mon implémentation C++ de P j'ai essayé de manipuler les actions aussi peu que possible et d'utiliser des références à la place. P renvoie un plan d'action partiellement ordonné A, R , avec A l'ensemble des actions et R l'ensemble des ordres entre les actions. Les instanciations d'actions dans mon implémentation C++ de P ne se produisent que lors de la construction de chaînes d'actions, afin de les insérer dans A, que je représente comme un vecteur d'actions. Enfin, R est un vecteur de vecteurs entiers et il stocke les ordres entre les références des actions dans A. Finalement, bien que je sois convaincu que la façon dont j'ai implémenté P ait joué un rôle dans ses performances, la qualité du processeur AMD Ryzen™ 7 2700X a également joué un rôle [50]. En particulier, le AMD Ryzen™ 7 2700X possède le Sense MI Technology, ce qui signifie que ce processeur utilise le Neural Net Prediction, un réseau neuronal dont le but est de comprendre les applications et d'anticiper les prochaines étapes du flux d'activité en temps réel. Cette technologie, combinée à de nombreuses autres, comme le pipelining, l'utilisation de répartiteurs, l'utilisation de threads, etc. peut également avoir joué un rôle.

• Il semble que de nombreux facteurs expliquent les énormes différences de performances entre P et P. De toute évidence, la réduction de la complexité a joué un rôle, surtout lorsque m et n sont élevés. Cependant, lorsque m est faible (< 100), la différence de performance est énorme et semble croître, c'est-à-dire que plus m est faible, plus la différence de performance est élevée. Les problèmes évolutifs mettent ce point en évidence. P est 10 fois plus rapide que P avec le benchmark MultiPrv_n_Cycle lorsque m est faible, et environ 2 fois plus rapide lorsque m est élevé. Avec le benchmark OnePrv_n_Cycle, P est 65 fois plus rapide que P lorsque m est faible, et 178 fois plus rapide lorsque m est élevé. Bien qu'il s'agisse de différences significatives dans les performances d'exécution entre les deux algorithmes, elles ne sont pas aussi significatives que celles des benchmarks réalistes (P est > 34.000 fois plus rapide que P). J'ai réalisé une "expérience AMD", c'est-à-dire que j'ai vérifié le comportement de mon processeur lors de la résolution des benchmarks réalistes, pour m'assurer que les plans n'étaient pas mis en cache. Ce n'était pas le cas, donc il peut y ce chapitre que la raison de la non-abordabilité des problèmes SAS-PU est liée aux opérateurs affectant la même variable d'état et bouclant ensemble. J'étudie donc de près et je restreins, avec C k , les cycles formés par les opérateurs bouclant ensemble. En particulier, je présente les classes de problèmes SAS-PUC 0 , SAS-PUC S 2 et SAS-PUC * 2 , et j'ai prouvé que ces problèmes sont abordables car leurs instances solvables sont résolues par des plans solution minimaux qui sont Υ 1 , c'est-à-dire qu'ils ne contiennent pas deux fois la même action. Finalement, je présente TopoPlan, noté P, un algorithme en temps linéaire qui résout les instances d'un problème SAS-PUC 0 , SAS-PUC S 2 ou SAS-PUC * 2 . J'ai prouvé, à l'aide d'un critère de vérité (Υ 1 -Truth Criterion), que cet algorithme est correct et complet. J'ai également prouvé que la complexité spatiale de P dans le pire des cas est de O(n 2 • m 2 ), c'est-à-dire quadratique par rapport au nombre de variables d'état et la taille maximale du domaine des variables. Enfin, j'ai prouvé que la complexité temporelle de P dans le pire des cas est de O(|O| + |E O |), c'est-à-dire linéaire en fonction du nombre d'opérateurs plus le nombre d'ordres entre les opérateurs. J'ai donc réussi à répondre à une partie de ma question principale, celle de savoir comment réduire la complexité temporelle.

Benchmarks and Results

Dans ce chapitre, je présente les performances d'exécution de P et P sur différents benchmarks. Je teste d'abord P et P sur des problèmes évolutifs, c'est-à-dire des problèmes dont la difficulté de résolution croît en fonction du nombre de variables d'état ou en fonction de la taille de chaque domaine de variables d'état. Ces repères me permettent de montrer empiriquement la complexité temporelle de P et Pİls montrent également que, globalement, P est plus rapide que P.

Ensuite, j'ai introduit des benchmarks réalistes : RDR-2, ACO et HZD, qui sont basés sur les jeux vidéo commerciaux Red Dead Redemption, Assassin's Creed : Origins et Horizon Zero Dawn respectivement. Ces benchmarks me permettent de montrer que mes classes de problèmes abordables sont applicables à des problèmes réalistes. En particulier, ils permettent de créer des problèmes qui n'étaient pas possibles avec SAS + -PUS et qui sont plus proches des problèmes du monde réel. Les performances d'exécution de P sur ces benchmarks sont impressionnantes, il est capable de produire des millions de plans en temps réel. Avec des instances adaptées qui peuvent être résolues à la fois par P et P je montre que P est beaucoup plus rapide que P pour les résoudre.

La création d'opérateurs reste cependant délicate avec mes classes de problèmes, certaines concessions peuvent être nécessaires pour rester dans les limites des restrictions. Le jeu en vaut cependant la chandelle, car les instances de ces problèmes sont résolues par P, mon algorithme en temps linéaire avec lequel j'ai prouvé qu'une fois optimisé, il est possible de planifier en temps réel pour des millions de PNJ. Finalement, la principale conclusion à la lumière des performances de P est que la planification en temps réel sur CPU est réalisable à grande échelle avec les technologies actuelles.

Travaux Futures

Dans cette section, je donne quelques idées pour poursuivre les recherches présentées dans mon manuscrit de thèse.

Figure 2 .

 2 2 shows in-game data representing 58 hours of playing F.E.A.R. over 617

Figure 2 . 2 :

 22 Figure 2.2:x-axis represents the action usage and y-axis is the action cost; the 72 (red) dots of this graphic represent the 72 actions logged in the F.E.A.R. sessions. We can observe that there is no clear influence of the action cost over the action usage: some, more expensive, actions are used much more than other, less expensive, actions.

Figure 2 . 3 :

 23 Figure 2.3: The 10 (red) dots of this graphic represent the 10 attack actions logged in the F.E.A.R. sessions achieved the same KillEnemy goal for the action set of the basic soldier and are the 10 actions from Figure 2.1;x-axis represents the action usage and y-axis is the action cost; We can observe that there is no clear influence of the action cost over the action usage: some, more expensive, attack actions are used much more than other, less expensive, attack actions.

Figure 2 . 5 :

 25 Figure 2.5: Probability of plans occurrence as a function of their length; 0 means GOAP has failed to build a plan.

Figure 2 . 6 :

 26 Figure 2.6: Comparison of the occurrence probabilities of the attack action plans between OC + , UC and OC -.

Figure 2 . 7 :

 27 Figure 2.7: Whisker box plots of the planning runtimes as a function of OC + , UC and OC -; the white mark represents the median value while the black mark represents the average value.

  (i.e. number of actions) µsec What is the runtime as a function of plans length?

Figure 2 . 8 :

 28 Figure 2.8: Comparison of planning runtimes with OC + , UC and OC -as a function of the plan length.

Proposition 2 . 4 . 1 (

 241 Working Assumptions). The following assumptions are compatible with the creation of realistic behavior for NPCs: 1. Actions are unary 2. Actions are contextually post-unique

Definition 3 . 1 . 1 (

 311 State description and world modeling).

Figure 3 . 1 :

 31 Figure 3.1: Read Dead Redemption based NPCs: the Horse Breeder.

Figure 3 . 2 :

 32 Figure 3.2: Read Dead Redemption based NPCs: the Native Hunter.

Definition 3 . 1 . 3 (

 313 SAS Operators). Let o ∈ O, post(o), pre(o) and prv(o) are functions that respectively return the post-, pre-and prevail-conditions of the operator o. These functions are defined as follows: pre(), post(), prv()

•

  S3: For all o ∈ O, dim(post(o)) and dim(prv(o)) are disjoint. • S4: For all o ∈ O and for all v i ∈ M, if pre(o)[v i ] = u, then pre(o)[v i ] = post(o)[v i ]. • S5: For all o, o ∈ O, if post(o) = post(o ) and (pre(o) prv(o)) (pre(o ) prv(o )), then o = o . • S6: For all o ∈ O, dim(pre(o)) = dim(post(o)).

Figure 3 . 3 :

 33 Figure 3.3: CubeWorld with 2 cubes. s 0 = 1, 0 and s = 0, 1

Figure 3 . 4 :

 34 Figure 3.4: A graphic parsing of PWEAK truth criterion.

Definition 4 . 1 . 2 .

 412 Let v i ∈ M and O(v i ) ⊆ O the set of unary operators affecting the state variable v i . For each o p v i ∈ O(v i ), I redefine the pre() and post() functions by taking into account the restrictions (P) and (U), and with respect to SAS structures (cf. subsection 3.1.3):

Table 4 . 2 :

 42 The different representation of the Horse Breeder operators: (a) This resemble the original representation except I take advantage of the restrictions (PU) to represent the pre-and post-conditions (Definition 4.1.2). (b) The sets of Neighbors (Definition 4.1.3). N pre is a singleton. As n = |D v 0 | = |D v 2 | = 3, operators can be indexed with the formula i × 3 + p. For instance, PickUpBucket index is 4. o 0 v 2 , the predecessor of o 1 v 2 via the precondition, is a ghost operator (Definition 4.2.3). (c) The set of clobberers (Definition 3.2.5).

Figure 4 .

 4 1 is Υ 2 , because the operators o + v 0 and o - v 0 are instantiated twice. Then, Lemma 3.2.1 can be rewritten: for each

1 .Corollary 4 . 1 . 1 .

 1411 ope(a) = o. If is made up of s 0 = pre(o) ⊕ prv(o) and s = post(o) ⊕ prv(o), then ∆ = a is the minimal solution plan solving and it is Υ 1 . It is possible to create such an instance in every SAS problem. It results the statement of Lemma 4.1.Every SAS problem is at least Υ 1 .

Figure 4 . 1 :

 41 Figure 4.1: The operator graphs G O = (O, E pre (O) E prv (O)) of the Horse Breeder and the Gray Code. The blue directed edges are post-pre dependencies from E pre (O) whereas the green directed edges are the post-prevail dependencies from E prv (O)

Figure 4 . 2 :

 42 Figure 4.2: All the domain operator graphs of the Horse Breeder and the Gray Code. The Horse Breeder has a ghost operator in G O (v 2 ), named Ghost V2, whose identifier is o 0 v 2 (cf. Definition 4.2.3).

we can identify exactly 2 operators with Defini- tion 4 . 1 . 1 ,

 411 namely o 1 and o 2 , such that post(o 1 ) = x 1 and post

  The minimal action graph G ∆,GC . The topological sort of G ∆,GC .

Figure 4 . 3 :

 43 Figure 4.3: Minimal action graphs solving the Gray Code instance:= (Φ GC , s 0 = 0, 0, 0 , s = 1, 0, 0 ). o + v 0 and o + v 0 are instantiated twice: ope(a v 0 ,+ ) = ope(a v 0 ,+ ) = o +

Theorem 4 . 4 . 1 .

 441 Let k ∈ N,

Corollary 4 . 4 . 1 .

 441 Let k ∈ N, SAS-PUC k ⊆ SAS-PU. Proof. Direct from Theorem 4.4.1 (1.) and Theorem 4.4.2.

2 Theorem 4 . 4 . 4 .

 2444 The class of problems SAS-PUB is included in the class of problems SAS-PUC 2 .

2

  Definition 4.4.2. A SAS-PU structure is restricted by C S2 iff:

Figure 4 .

 4 5 respectively shows the declobbered operator graph of the Horse Breeder and that of the Gray Code. I use these graphs to increment the restriction C 2 as (P), (U) and C 2 are not sufficient to ensure a tractable class of problems (cf. Corollary 4.4.2). I denote C * 2 the incremented version of C 2 . Definition 4.4.6 (Restriction C * 2 ). Let v i ∈ M, x, y ∈ D v i , x = y. • The class of problems is C 2 The Horse Breeder declobbered operator graph (G + O HB ). The Gray Code declobbered operator graph (G + O GC ).

Figure 4 . 5 :

 45 Figure 4.5: The yellow nodes are requestable operators inside the unique cycle of the same domain operator graph. The blue and green nodes are operators preceded by a different yellow operator via a post-prv dependency. The blue edges represent the post-pre dependencies, the green edges the post-prv ones and the red edges represent orders by promotion (cf. Theorem 4.3.1).

  Horse Breeder: G O\Cycle(v 1 ) HB . Gray Code: G O\Cycle(v 0 ) GC .

Figure 4 . 6 :

 46 Figure 4.6: G O\Cycle(v i ) are declobbered operator graphs with the domain operator graph on v i removed. It is useful to check whether green and blue operators are related in it. Here are two examples: the Hose Breeder and the Gray Code.

  Figure C.4 efficiently shows how a planner can unfold many times the yellow cycle of a simple SAS-PUC 2 structure (C.4a) in order to provide a minimal and correct non-linear solution plan (C.4c). Figure C.3 also shows a SAS-PUC 2 problem where instances, except one (C.1c), have a minimal solution plan that is Υ 1 . The exception problem instance has a minimal solution plan that is Υ 2 . Eventually, Figures C.1 and C.2 show two SAS-PUC *2 problems and several minimal and Υ 1 solution plans that solve different problem instances. These two examples illustrate prominently the effect of the restriction. The non-relationship between o 1 and o 2 , or between the "green" and the "blue" operators, (cf. Definition 4.4.6 and the figures), implies that the instance of the requestable operators can frame once all the instance of the green and blue operators.

Theorem 4 . 4 . 6 . 1 . 4 . 4 . 5 3 Lemma 4 . 4 . 1 .

 44614453441 , C.2d and the left graph of C.2e, they are all ordered after a + ), or can also be ordered right after the start if p = s 0 [v i ] (For examples, see the blue actions of Figures C.2f and the right graph of C.2e). SAS-PUC * 2 problems are Υ The proof is provided in Appendix C.2. The Class of Problems SAS-PUC k , with k ≥ SAS-PUC k problems with k ≥ 3 are at least Υ 2 .

Figure 4 . 7 :

 47 Figure 4.7: SAS-PUCk: The inclusions.

Figure 4 . 8 :

 48 Figure 4.8: .txt file of the Horse Breeder.

Theorem 4 . 5 . 3 (Procedure 2

 4532 Time Complexity). P worst-case time complexity is O(|A| + |E A |), withA the set of actions and E A the set of orders between the actions of A. Proof. Phase 1 has at most O(|O|) = O(mn) steps: m steps via the for loop (3.2) times n steps via the BuildChain procedure of (3.4). Phase 2: Due to the restriction Υ 1 , |A| ≤ |O|, so the for loop (3.9) requires at most |A| ≤ |O| = O(mn) steps. The two for loops (3.9) and (3.10) require O(|A| + |E A,prv |) steps with E A,prv = {(b, a) | a ∈ A ∧ b ∈ N prv (a)}. Indeed, during this step, the two for loops check the prevail-conditions of the whole set of actions of A: there are thus |A| steps to go through all the actions of A, plus |E A,prv | steps to visit all the predecessors of all the actions of A. The procedures BuildChain (3.15) and (3.21) can be triggered if the action being evaluated has a prevail-condition that is neither satisfied by the initial state nor by an action of A. In this case, both procedures look for actions that are in a set D ⊆ O \ A. The procedures BuildChain thus add |D| steps for their calls. The prevail-conditions of these newly added actions in D must be checked as well, it thus adds |D| more steps via the for loop (3.9) as well as |E D,prv | more steps for the for loop (3.10). Finally, phase 2 has a complexity of O(|A|+|E A,prv |+2•|D|+|E D,prv |) ≡ O(|A|+|E A,prv |+2•|O\A|+|E O\A,prv |) ≡ O(|O|+|E O,prv |). All the instructions from (3.27) to (3.30) are done in constant time (O(1)). Phase 3: At the end of Phase 2 is returned ∆ = (A, E A ), a non-linear sequence of actions. Phase 3 topologically sorts ∆ and its worst-case time complexity is O(|A| + |E A |); in the worst case, this is equivalent to O(|O|+|E O |), which ultimately dominates the whole and proves the linear time complexity of P. BuildChain(v i , s, g, A, E A , O) Input/Output: cf. Definition 4.5.2. Parameters: x, y, two values of

33 :

 33 if Color(a) = yellow then34: DFSTopo(a, A, E A , s 0 , ∆)35: end if 36: end for 37: return ∆ Theorem 4.5.4 (Space Complexity). P requires at most O(n 2 • m 2 ) space.Proof. A (PU) operator takes O(m) space: the pre-and post-condition can be reduced to one variable each, plus a variable for the index of the affected state variable (cf. Definition 4.1.2), and the set N pre is a singleton due to restrictions (P) and (U) ; the prevailconditions, on the contrary, are lists of m elements, and the set N prv has at most m elements (cf. Definition 4.1.3). Then, the hashing table that stores identifiers (cf. Definition 4.1.1) takes O(n • m) space. Finally, P creates an action graph G ∆ = (A, E A ) which takes at most O(n 2 • m 2 ) space: The set of actions takes O(|A|) ≡ O(|O|) ≡ O(n • m) space due to Υ 1 , and each action can have at most O(|A|) ≡ O(n • m) predecessors. Hence, O(|A| 2 ) ≡ O(n 2 • m 2 ), which dominates the whole.It should be noticed that, if an action C ∈ A has |A| -1 predecessors, it is the following scenario: C is the clobberer of all the actions of the problem, except for the establisher, and these actions have been promoted. It results that C is preceded by the establisher and by all the promoted actions, hence O(|A|). Another consequence is that the other actions only have one predecessor: the establisher. In most problems, the number of predecessors of an action a is closer to O(|N prv (a)|) ≡ O(m) as it corresponds to the number of establishers to assert all the prevail-conditions of a. And the clobberers, which are the successors of the establishers, are ordered after a (most of the time, by promotion, cf. subsection 4.3.2), i.e. each clobberer only gets one additional predecessor: a. Overall, there are at most O(n • m 2 ) orders between actions, but a single action can have up to O(|A|) predecessors.

Figure 5 . 1 :

 51 Figure 5.1: Tunnel example.

Figure 5 . 2 :

 52 Figure 5.2: P can run in O(m) when |A| + |E A | = O(m) whereas P always runs in O(m 2 ).

Figure 5 . 3 :

 53 Figure 5.3: Runtime ratio between the two algorithms ( P P ), with respect to the scalable problems of MultiPrv_2_Cycle and OnePrv_2_Cycle.

  (n) for each of the m state variables of a Multi_n_Cycle instance Total runtime



                                      • Data from  Data (n) for each of the m state variables of a OnePrv_n_Cycle instance Total runtime (millisecond) (b) OnePrv_n_Cycle.

Figure 5 . 4 :

 54 Figure 5.4: Comparison of the runtime performances of P and P when solving MultiPrv_n_Cycle or OnePrv_n_Cycle in function of n and with three fixed a m ∈ {2, 4, 8}.

  (n) for each of the m state variables of a OnePrv_n_Cycle instance Runtime Ratio  ℙ (b) OnePrv_n_Cycle.

Figure 5 . 5 :

 55 Figure 5.5: Runtime performance ratio of P over P when solving MultiPrv_n_Cycle or OnePrv_n_Cycle in function of n and with a fixed m ∈ {2, 4, 8}.

Figure 5 . 6 :

 56 Figure 5.6: BlockWorld. s 0 = 0, 0, 1, 0, 0, 1 and s = 1, 0, 0, 1, 0, 0 . M = {AonB, AonC, BonA, BonC, ConA, ConB}.

Figure 5 . 8 :

 58 Figure 5.8: The operator graph of the Horse Breeder. The blue directed edges are the post-pre dependencies whereas the green directed edges are the post-prevail dependencies.

  (a) The declobbered operator graph of the Horse Breeder with red directed edges representing orders by promotion (Definition 3.2.5). If one red edge is added between two actions by Pwhile planning, it means the queue action clobbers a prevail-condition of the head action. If the Horse Breeder needs to Drop Haystack to Pick up Bucket, then the Take Haystack action, if required, threatens the prevail-condition of the action Pick up Bucket. Therefore, to ensure a proper topological sort, the add of the directed red edge from Pick up Bucket to Take Haystack by Pwhile planning is necessary. The declobbered operator graph of the Horse Breeder when the operators Pick up Bucket and Drop Bucket are removed (yellow operators in Figure 5.9a).

Figure 5 . 9 :

 59 Figure 5.9: This two subfigures highlight why the Horse Breeder problem respects the restriction C * 2 . "Yellow"= {Pick up Bucket, Drop Bucket}, "Green"= {Fill Bucket with Water, Fill Horse Trough} and "Blue"= {Take Haystack} are affected by the restriction C * 2 . It is C 2 because the "Yellow" nodes form a cycle of size 2 in their domain operator graph (cf. Figure 5.7). It is C *2 because none of the "Green" nodes are linked to the "Darker Green" one when the "Yellow" nodes are removed (cf. Figure5.9b).

v 4 :

 4 Debating, D v 4 = {0 : f alse, 1 : true}. v 5 : isRested, D v 5 = {0 : f alse, 1 : true}. v 6 : W orking, D v 6 = {0 : f alse, 1 : true}.

Forge a Saber v 7 :

 7 rawM etals, D v 7 = {0 : stored, 1 : inF oundry}. v 8 : smeltedM etal, D v 8 = {0 : canBeCollected, 1 : collected}. v 9 : f orgedSword, D v 9 = {0 : f alse, 1 : true}. v 10 : f orgedSaber, D v 10 = {0 : f alse, 1 : true}.

Figure 5 . 10 :

 510 Figure 5.10: All the domain operator graphs of my representation of Assassin's Creed: Origins. These graphs present the post-pre dependencies between operators. Ghost operators are operators that can be identified with the pair (v i , p) (Definition 4.1.1) but do not exist in the problem (Definition 4.2.3).

ForgeSaberFigure 5 . 11 :ForgeSaberFigure 5 . 12 :

 511512 Figure 5.11: The operator graph of an armorer citizen of Assassin's Creed: Origins. The post-pre dependencies are in blue whereas the post-prevail dependencies are in green.

ForgeSaberFigure 5 . 13 : 2 .

 5132 Figure 5.13: The graph of operators of Assassin's Creed: Origins, except that the operators Work and End Work are removed. This graph is here to illustrate the structural restriction C * 2 . The operators Work and End Work are in Cycle(v 6 ) and both satisfy the v 6 prevailcondition of some other operators: Work satisfies the v 6 prevail-condition of "Green"= {Put Metal in Foundry, Collect Smelted Metal, Forge a Sword, Forge a Saber}, whereas End Work satisfies the v 6 prevail-condition of "Darker Green"= {Buy Food, Buy Drink, Take, Store, Have a Walk, Rest}. No green operator is connected to any of the darker green operators, therefore the structural restriction of C * 2 is respected (Definition 4.4.6).

Figure 5 . 14 :

 514 Figure 5.14: The operator graph of the A-M model, with ghost operators removed. The blue directed edges are the post-pre dependencies whereas the green directed edges are the postprevail dependencies. There is no clobbering situation in this model (Definition 3.2.5), therefore there are no red directed edges like in the models of RDR-2 and ACO.

Figure 5 . 15 :

 515 Figure 5.15: The domain operator graphs of the Acquisition Machine (A-M) model. Each graph represents the post-pre dependencies between some operators. Ghost operators are operators with an existing identifier (cf. Definition 4.1.1) but that do not exist in the problem.

Figure 5 . 16 :

 516 Figure 5.16: The performance of Pwhen solving HZD (blue square) and the SAS-PUC * 2

Figure 5 . 18 :

 518 Figure 5.18: The performance of P when solving HZD (black diamond) and the SAS-PUSC 2 versions of RDR-2 and ACO (black triangles and dots resp.)

  x |M| avg(|E A |) Performances of P HZD (SAS-PUSC 0 ) Fig. 5.16 and 5.17

  0

Figure 5 . 19 :

 519 Figure 5.19: Inner analysis of the AMD Ryzen™ 7 2700X, on how the C++ implementation of P is processed during the experiment. This graph shows the number of Retired Instructions (RI) and the Instruction Per Cycle (IPC) wrt. the number of NPCs. The left x-axis is for the number of RI, the right x-axis is for the IPC, and the y-axis is the number of NPCs. The solid curves show the RI whereas the dashed curves show the IPC, both wrt. TopoPlan, the name of our algorithm; and wrt. SearchExtension, BuildChain and DFSTopo, functions (methods) used by TopoPlan and Init, the method used to initialize the start and the goal states for each problem instance.

6 Conclusion 6 . 1

 661 How to Reduce Time Complexity to Scale up Realtime Planning?

Figure B. 1 :

 1 Figure B.1: The Mathematica source code to produce the domain operator graphs presented in the appendix, or with higher size.

Figure B. 5 :CRestrictions C 2 , C S 2 and C * 2 I

 522 Figure B.5: |D v i | = 5. All possible shapes of domain operator graphs with 5 operators. The number on the left of each graph is an identifier for referencing.

2 δ

 2 vi (+,s * ) δ vi (-,s * ) (a) Operator Graph.

2 δ

 2 vi (+,s * ) δ vi (-,s * ) (b) Declobbered Operator Graph.a1 a2

Figure C. 1 :

 1 Figure C.1: A SAS-PUC * 2 structure: o 2 is not related to o 1 when the yellow part of the declobbered operator graph is removed. Let v i ∈ M, (C.1c) to (C.1f) are Υ 1 minimal solution plans of problems s.t. s 0 = s 0 [v i ], pre(o 1 ), pre(o 2 ) and s = s [v i ], post(o 1 ), post(o 2 ) .The problems differ with s 0 [v i ] and s [v i ] defined in the sub-caption. Finally, we have:pre(o + v i ) = post(o - v i ) = -; pre(o - v i ) = post(o + v i ) = + ; prv(o 1 )[v i ] = + and prv(o 2 )[v i ] = -.

6 (

 6 b) Declobbered Operator Graph.

Figure C. 2 :

 2 Figure C.2: Let v i , v j ∈ M. (C.2c) to (C.2f) represent graphically non-linear, Υ 1 and minimal action plans solving problems such that, for j ∈ [[1, 6]], s 0 [v j ] = pre(o j ) and s [v j ] = post(o j ). Instances of o j are denoted a j . The problems only differ via s 0 [v i ] and s [v i ], and their respective definition is given in each sub-caption. We have pre(o+ v i ) = post(o - v i ) = -, and pre(o - v i ) = post(o + v i ) = +, and a + and a -are resp. the instance of o + v i and o - v i . Finally, the blue operators/actions have +, and the green ones have -, as a prevail-condition on v i . (C.2c) and (C.2e) have two minimal, Υ 1 and non-linear solution plans. It should be noted that, as for the Horse Breeder structure, green (resp. blue) operators can be related between each others.

2

  ov i ,+ ov i ,- o1 o2 δv i (+,s*) δv i (-,s*) (a) Operator Graph.

2 δ

 2 vi (+,s * ) δ vi (-,s * ) (b) Declobbered Operator Graph.

Figure C. 3 :

 3 Figure C.3: A SAS-PUC 2 structure: o 1 is related to o 2 in the declobbered operator graph when the yellow part is removed, via the dashed green directed edge. The notation is equivalent to the previous Figures. The minimal and non-linear solution plan of Figure C.3c is Υ 2 .In this problem, a 2 must be demoted after a + because a + establishes a 1 which itself establishes a 2 . Then, v i = -is not the goal state, so a 2 is promoted before a + ' which satisfies the pre-condition ofδ v i (+, s [v i ]). Finally, δ v i (+, s [v i ]) satisfies s [v i ].

6 (

 6 Figure C.4: A SAS-PUC 2 structure: each blue operators is related to one or two green operators via the blue directed edges, even when the yellow nodes are removed from the declobbered operator graphs. I just provide one problem to illustrate how a cycle of a domain operator graph can be expanded in such a structure. The solution plan of Figure C.4c is Υ 3 : o + v i and o - v i have both been instantiated three times to provide a minimal and non-linear solution plan to the problem.

v 25 :

 25 isRelaxed, D v 25 = {0 : f alse, 1 : true}. v 26 : endDayRoutine, D v 26 = {0 : notStarted, 1 : bodyW ashed, 2 : Rested}. v 27 : workRoutine, D v 27 = {0 : notStarted, 1 : readyT oW ork, 2 : hasDanced, 3 : hasW orked}.Table F.3: Operator set of the Dancer. His role is to entertain the crowd, the audience: s (Work) = 0, 0, 0, 3 . The Dancer is a SAS-PUSC 0 problem, solvable by both Pand P.

O

v 4 : 5 :

 45 HorseHarnessed, D v 4 = {0 : f alse, 1 : true}. v HorseF oot, D v 5 = {0 : oldShoes, 1 : cleaned, 2 : shoesOf f, 3 : rubbed, 4 : newShoes}. v 6 : F orgeIsLit, D v 6 = {0 : f alse, 1 : true}. v 7 : Hammer, D v 7 = {0 : stored, 1 : equiped}. v 8 : inF orge, D v 8 = {0 : nothing, 1 : coldShoes, 2 : hotShoes}. v 9 : Shoes, D v 9 = {0 : old, 1 : shaped}.

  HuntingRoutine, Dvi[10] = {0 : notStarted, 1 : onGoing, 2 : done}. (v 11 )HuntingReadiness, Dvi[11] = {0 : notReady, 1 : Ready}. (v 12 )Energy, D v 12 = {0 : low, 1 : rested}. (v 13 )T hirst, D v 13 = {0 : thirsty, 1 : hydrated}.

G. 1

 1 Introduction La Planification en Temps-Réel : Un problème de planification d'IA, ou plus précisément un problème de planification d'action d'IA (à distinguer du pathfinding et d'autres sous-domaines de la planification), est un problème d'IA où un agent intelligent peut planifier dans son environnement pour atteindre ses objectifs. Pour ce faire, l'environnement et les paramètres pertinents de l'agent sont symbolisés pour former des états que l'agent utilise pour évaluer une situation réelle à un moment donné de manière abstraite. La plupart du temps, un agent évalue sa situation actuelle ou ses objectifs. L'agent est ensuite équipé d'actions ou d'opérateurs que le planificateur utilise pour créer des plans d'action afin de se projeter dans le temps et d'effectuer des actions dans le monde. De plus, les actions et les plans d'action agissent tous deux comme une fonction de transition entre deux états. Ils peuvent avoir un coût afin de caractériser la qualité d'une transition, c'est-à-dire la pertinence d'une action ou d'un plan d'action pour une situation donnée.

Definition G. 4 . 2 (

 42 Dans cette section, je présente des notions et définitions qui tirent avantage des restrictions (P) et (U). En particuliers, je rappelle ce qu'est un opérateur sollicitable : Definition G.4.1. Soit RO l'ensemble des opérateurs sollicitables (requestable operator) de O. Ces opérateurs résolvent une prevail-condition d'au moins un autre opérateur de O.RO = {o | v i ∈ M, o ∈ O(v i ), post(o)[v i ] ∈ R O v i }Les notations Υ 1 et Υ 2 se généralise de la manière suivante : Restriction de sortie Υ k ). Soit k ∈ N, un plan d'action ∆ est Υ k ssi:

Theorem G. 4 . 1 (

 41 Υ 1 -Truth Criterion). Soit ∆ = (A, ≺) un plan solution non-linéaire et Υ 1 d'une instance de problème SAS-PU = (Φ, s 0 , s ). Soitσ v i = (A(v i ), ≺) une chaîne d'actions qui affecte v i ∈ M. ∆ est correct et Υ 1 ssi:Off requis par un autre opérateur B. Tous les opérateurs sont concernés par la restriction (C * 2 ) et il faut enlever les deux commutateurs l'un après l'autre de l'Operator Graph pour voir si (C * 2 ) est respecté. Il n'est pas respecté parce qu'il y a un chemin entre A et B et un chemin entre B et A dans l'Operator Graph après avoir retiré un commutateur. Dans une telle situation, l'astuce consiste à fusionner les deux commutateurs en un seul. C'est ce que j'ai fait avec le Native Hunter, avec les opérateurs Unequip et GetReadyToHunt. Initialement, j'ai fourni au Native Hunter les opérateurs EquipBow, UnequipBow, EquipQuiver, UnequipQuiver. Pour GoHunting, le Native Hunter devait EquipBow et EquipQuiver et, ensuite, pour StoreTheGame, le Native Hunter devait UnequipBow et UnequipQuiver. Ce modèle est équivalent à la situation des deux commutateurs On/Off décrite précédemment et ne respecte pas (C * 2 ). Par conséquent, EquipBow et EquipQuiver ont été fusionnés en GetReadyToHunt et UnequipBow et Un-equip Quiver ont été fusionnés en Unequip. (C * 2 ) est donc maintenant respecté. Finalement, j'ai décidé de créer une version SAS-PUSC 2 en intervertissant l'ordre entre Store the Game et Unequip. Initialement, le Native Hunter devait Unequip avant StoreTheGame, dans la version finale c'est finalement l'inverse.

Table 2 .

 2

1: F.E.A.R. Data from my experiments only consider the successful plans.

Table 3 . 1 :

 31 Horse Breeder operator set. His goal is to feed the horses: s * = 2, 0, 2 .

Table to A

 to Table 3.2: BlockWorld operator set.

	The BlockWorld problem is (P), (U), (B) and (S). It is (P) because no two operators
	affects the same post-condition, (U) because each operator only has one defined post-
	condition. It is (B) because |D AonB | = |D BonA | = 2, i.e. the domains of AonB and BonA
	are binary. Finally, it is (S) because, for each

Table 4

 4

	Operators	pre-	post-	prevail-
	o

.1 shows a problem of a 3-bits Gray Code where o + v 0 is instantiated twice. With a 4-bits Gray Code and a similar problem instance, o + v 0 would have been instantiated 4 times. With 5-bits, 8 times and so on. The operators of the Gray Code problem can be instantiated many times because operators

  which implies o n+1 is necessary preceded by an operator of (G O (v i )) n . This therefore adds a new node o n+1 and a new directed edge from (G O (v i )) n to o n+1 . The new directed edge does not create another cycle as x n+1 / ∈ (D v i ) n . Therefore, (G O (v i )) n+1 has a unique cycle and no ghost operator. It concludes the point 2. of the lemma: If there is no ghost operator, G O (v i ) has a unique cycle.

  a 1 , a 2 and a 3 be the action instances of the operators o x v i , o y v i , o z v i , o 1 , o 2 and o 3 respectively, we have:

  a 1 , a 2 and a 3 the action instances of the operators o x v i , o y v i , o z v i , o 1 , o 2 and o 3 , respectively, from Figure 4.4, to illustrate Table 4.3. We have:

  Concerning the uniqueness of C, there is only one case (cf. case (3)) where there are 2 clobberers, otherwise, cases (1) and (2) prove that C, if it exists, is simply unique. Concerning case (3) and its two clobberers C 1 and C 2 , because a chain of actions is linear, only C 1 clobbers the start (p = s 0 [v i ] = pre(C 1 )) while only C 2 clobbers the post-condition of the establisher T (p = post(T) = pre(C 2 ), i.e. T ≺ C 2 ). Although C 1 and C 2 clobber the same proposition p in case (3), ordering a either in function of C 1 or in function of C 2 is sufficient because the ordering in function of the second will be set automatically by transitivity via

	to the class SAS-PU with Definition 3.2.5 and the equations (4.3)
	to (4.9). The establishment process (T ≺ a) is proven by the equations (4.3), (4.5), (4.7)
	and (4.9). The promotion process (a ≺ C) is proven by the equations (4.3) (4.4), (4.6),
	(4.8) and (4.9).

  O) a structure of the class SAS-PUC k-1 . Let C the set of cycles having at least one requestable operator, then max

c∈C (|c|) ≤ k -1. Φ is therefore also of the class SAS-PUC k , because max c∈C (|c|) < k. Hence, SAS-PUC (k-1) ⊂ SAS-PUC k .

2. The restriction C k restricts the number of (PU) operators inside non-empty Cycle(v i ) having at least one requestable operator. So, with (S), C k merely restricts the number of operators inside Cycle(v i ) having exactly one requestable operator. Hence, SAS-PUSC k ⊂ SAS-PUC k .

Table 4 .

 4 it results that SAS-PUB ⊆ SAS-PUC 2 . The class of problems SAS-PUC 2 is intractable.Proof. The class of problems SAS-PUB is intractable because there exists problems such that the minimal solution plan is of length Θ(2 |M| ) (Theorem 6.14[6, p.138]). As SAS-PUB is included in SAS-PUC 2 (Theorem 4.4.4), the latter is intractable for the exact same reason. The class of problems SAS-PUC k , with k ≥ 2 is intractable. Proof. Let k ∈ N, Theorem 4.4.1 implies SAS-PUC 2 ⊂ SAS-PUC k and SAS-PUC k is therefore intractable according to Corollary 4.4.2. 4 presents the operator set of a SAS-PUC 2 problem:

	Corollary 4.4.2. Corollary 4.4.3. O	pre() post()	prv()

Table 4 .

 4 2(c) illustrate N clob (o p v i ) with the Horse Breeder. Operator graphs can then be enhanced with a set of directed edges that represents the promotion orders. + O the directed edge set. The declobbered operator graph is an operator graph where orders from the promotion process are displayed. E + O = E pre (O) E prv (O) E clob (O) with E pre (O) and E prv (O) the edge sets of Definition 4.2.1 and E clob

	Definition 4.4.5 (Declobbered Operator Graph).
	Let o p v

i ∈ O, I denote G + O = (O, E + O ) the declobbered operator graph, with O the vertex set and E

  be a ghost action.} then

	6:	Next(a y v i ) ← a g v i
	7: end if
	8: while y = s do
	9:	if a y v i is a ghost action then fail
	10:	end if
	11:	if Color(a y v i ) = yellow then fail
	12:	end if
	13:

  if A = ∅ then return ∅ {s 0 and s are equal.}

	5:	end if
	6: end for
	7:

8: end if 9: for a p v i ∈ A do {Phase 2} 10:

Table 5 .

 5 CfromTtoB 3: The operator set to solve problems of the BlockWorld with 3 blocks. m = 6, n = 2, |Oset| = 12. This representation, however, does not allow the displacement of a block from a block to another, each block displacement is from one block to the table and from the table to a block. A valid plan for the problem instance rev is therefore to move every block on the table and then build the goal tower.

	#(Blocks) |M| Runtime P(µs) Runtime P(µs) Ratio P/P
	2	4	0.1	275	2750
	3	6	0.1	636	6360
	10	90	3.40	678	199.4
	25	600	21.0	1.15 • 10 3	54.7
	50	2450	88.6	8.99 • 10 3	101.4

Runtime performance of P on the BlockWorld: The BlockWorld problem is interesting to illustrate some drawbacks of the SAS-PU representations. This BlockWorld problem is of the class SAS-PUC S 2 and therefore solvable by P.

Table 5 .

 5 4: Runtime performance of both P and P on the scalable BlockWorld.

Table 5 .

 5 5: Operator set of the Horse Breeder. His goal is to s (Feed the horses) = 2, 0, 2 . The Horse Breeder is a SAS-PUC * 2 problem, solvable by Ponly.

  Fill Bucket with Water and Fill Horse Trough have a different prevail-condition on v 1 than the operator Take Haystack. It is C * 2 , however, as it can be seen with Figure5.9. (i) The prevail requestable operators, Pick up Bucket and Drop Bucket, colored in yellow in Figure5.9a, are inside Cycle(v 1 ) and |Cycle(v 1 )| = 2 (cf. Figure5.7). (ii) Then, when these operators are set apart, the resulting graph is Figure5.9b, and we observe that Take Haystack is not linked to either Fill Bucket with Water or Fill Horse Trough. (i) and (ii) ensure the restriction C * 2 is respected. The domain operator graphs of the Horse Breeder. These graphs represent the post-pre dependencies only. Ghost V2 is a ghost operator, i.e. an operator whose identifier exists but the operator itself does not exist (Definition 4.2.3).

	Ghost V2	Fill Bucket with Water	Fill Horse Trough
	Drop Haystack	Take Haystack	Fill Horse Feeder
	Pick up Bucket	Drop Bucket
	Figure 5.7: Drop Haystack Ghost V2
	Fill Bucket with Water		Fill Horse Feeder
		Pick up Bucket
			Take Haystack
	Fill Horse Trough	Drop Bucket
			2; ids={2,4} B.3;
	ids={2,6,7,9} B.4; etc.) or a full cycle (cf. appendices B: id=1 B.2; id=3 B.3; id=8 B.4;
	etc.)
	This Horse Breeder problem is obviously (P) and (U). It is not (S), however, because

Table 5 . 6 :

 56 Possible operator representation for the citizens of Assassin's Creed: Origins (based on the game)

Table 5 .

 5 7: Example of a detailed NPC for Assassin's Creed: Origins (Armorer).

Table 5 .

 5 10: This table shows the performances of Pand Pin the light of the linear combinations of the number of NPCs (x) constructed from the data presented in the Figures 5.16, 5.17 and 5.18. The performances of Pare sorted in descending order of the average number of orders between actions (avg(|E A |)). The performances of Pare sorted in descending order of the number of state variables (|M|).

	.0676405 + (3.7637 • 10 -7 )x	7	12.6667

Table 5 .

 5 11: Characteristics of the AMD Ryzen™ 7 2700X used for the experiments.

end for 10: for h ∈ H do 11:

  {a} 10: if C = then Input: M, set of state variables; H, set of operators; D v 1 , ..., D v |M| , Domains of state variables ; s 0 and s , start and goal states respectively. Parameters: A, T , sets of actions; H V hashing table of operators; R, set of relations; R 1 , R 2 , boolean matrices; F , boolean list; ᾱv i , βv i , β , γv i , chains of actions. Output: A, R , non-linear action plan. Tuple with A the set of actions and R the relations between them. : A ← ∅; T ← ∅; R ← ∅ 2: R 1 ← size |M| × 2.|H| boolean matrix. All entries set to false 3: R 2 ← size 2.|H| × |M| boolean matrix. All entries set to false 4: F ← size |M| boolean list. All entries set to false 5: H V ← size |M| vector of vector pointers 6: for v i ∈ M do H V [v i ] ← size |D v i | vector of action types with entries initialized to ∅ 9:

	7:	ᾱv i ← ; βv i ← ; γv i ← ;
	8:
	11:	Order Last(C) before a in R
	12: end if
	13: return C ⊕ a

1

  boolean vector. All entries set to false 15: for v i ∈ M do

	16:

  |D v i | = 3. All possible shapes of domain operator graphs with 3 operators. The number on the left of each graph is an identifier for referencing. |D v i | = 4. All possible shapes of domain operator graphs with 4 operators. The number on the left of each graph is an identifier for referencing.

	1,	, 2,	, 3,	, 4,			
	Figure B.3: 1, 4, 7, 5, 9, 13, , 14, , 2, , 6, , 10, Figure B.4: 1, 17, , 18,	, 2, , 5, , 8, , 15, , 3, , 7, , {11, , 19,	, 3, , 6, , {9, , 16, , 4, , , 8, }, {12, , {20,	, }	, }	, ,	},

  +}. Such problem instances are equivalent to solving an instance of a SAS-PUC S 2 problem with k cycles affected by the restriction C S 2 . So the solution plan, if it exists, is necessarily Υ 1 according to the induction hypothesis. If we consider s0 [v 1 ] ∈ {-, +} and s [v 1 ] ∈ {-, +, x}, the different forms of the chain of actions chain v k+1 follow the same explanation as in P (1). It results that the only chain of actions that could have been at least Υ 2 is not minimal if not Υ 1 . Eventually, P (k + 1) is proved and it concludes the proof.

	C.2 Proof of SAS-PUC * 2
	Theorem C.2.1. SAS-PUC * 2 problems are Υ 1 .

Proof. Every SAS problem is at least Υ 1 (Lemma 4.1.1). I prove that SAS-PUC * 2 problems are at most Υ 1 by recursion. Base case: Let consider any SAS-PUC * 2 problem instance (1) = (Φ (1) , s 0 , s ) with Φ (1) a SAS-structure that has only one cycle restricted by C * 2 . v 1 ∈ M, let denote Cycle(v 1 ) that single cycle restricted by C * 2 , we have RO(v 1

  This chain is then concatenated with smaller sequence of actions that are minimal and Υ 1 because they solve smaller problem instances that are either from SAS-PUC 0 problems (Theorem 4.4.3), or from SAS-PUC S 2 problems (Theorem C.1.1), or from SAS-PUC * 2 problems with exactly k cycles restricted by C *

2 (Induction hypothesis).

This concludes the proof, SAS-PUC * 2 problems are Υ 1 .

  be noticed that a |D|+1 is either (1) a ghost action, or (2) it is white (i.e. not in a chain of actions), or (3) it is yellow. (1) and (2) can be used by the procedure DFSTopo as a valid stop condition to break the recursivity (2.3), because, in cases (1) or (2), a |D| is obviously the first action to modify the start on v i . In case (3), then the additional order acts as a concatenation between two minimal chains. Case (3) means a |D|+1 belongs to another action chain than chain v

i (s, g). a |D|+1 is necessarily the last action of this other chain. Indeed, due to the coloring system it cannot exist two different chains of actions with similar actions in them (cf. Lemma D.1.2). Therefore, a |D|+1 is the last action of a chain on v i and a |D| is the first action of chain v i (s, g), it concludes the fact the additional order (a |D|+1 , a |D| ) acts as a concatenation between two chains on v i .

It results that the procedure BuildChain could have had D and E + D as local parameters so that the instructions:

•

  It means P (3.10) (r) holds for at least one r ∈ [[1, |N prv (a p v i )|]], then P (3.10) (r) holds for every r ∈ [[1, |N prv (a p v i )|]] by independence of each loop. Then |N prv (a p v i )| < |M|, so the loop (3.10) always terminates. Then, it may exist actions with no defined prevail conditions, however (cf.

  BuildChain builds backwards (3.4):

					F
				Red Dead Redemption 2: Operator Sets
	O	Pre	Post	Prevail v 14 , v 15 , v 16 , v 17 , v 18 Explanation
	o 1 v 14 o 2 v 14	v 14 = 0 v 14 = 1	u, u, u, u, u	Receive / Give the orders

  2, 1, u Attack the Bank v 14 : AttackReadiness, D v 14 = {0 : none, 1 : OrdersGiven, 2 : inP osition}. v 15 : Rif f le, D v 15 = {0 : stored, 1 : equiped}. v 16 : Ammo, D v 16 = {0 : none, 1 : inHandbags, 2 : engaged}. v 17 : BanditBag, D v 17 = {0 : dropped, 1 : equiped}. v 18 : BankAttacked, D v 18 = {0 : f alse, 1 : true}. Table F.1: Operator set of the Bandit. His goal is to s (Rob bank) = 2, 1, 2, 1, 1 . It can be noticed that the operator Receive/Give the Orders represents in fact two operators. The leader Give the Orders while the rest of the group Receive the Orders. These actions can get a parameter for contextualization: Give the Orders(?toNPC), Receive the Orders(?fromNPC). The Bandit is a SAS-PUSC 2 problem, solvable by both Pand P.

	O	Pre	Post	Prevail v 28 , v 29 Explanation
	o 0 v 28

  2 1, u Drink v 27 : P osition, D v 27 = {0 : Home, 1 : Bar}. v 28 : Drink, D v 28 = {0 : none, 1 : hasOne, 2 : hasDrunk}. Table F.2: Operator set of the Drunk Citizen. He is just roaming in the streets: s (Forget) = 0, 2 . The Drunk Citizen is a SAS-PUSC 2 problem, solvable by both Pand P.

	O	Pre	Post	Prevail v 23 , v 24 , v 25 , v 26 , v 27 Explanation
	o 1 v 23

Table F . 4 :

 F4 Operator set of the Farrier. His goals are s (Shoeing the horse) = 0, 0, 4, s 0 [v

6 ], s 0 [v 7 ], s 0 [v 8 ], s 0 [v 9 ] and s (Forge horseshoes) = 0, s 0 [v 4 ], s 0 [v 5 ], 1, 0, 2, 1 .

The Farrier is a SAS-PUC * 2 problem, solvable by Ponly.

Table F .

 F 5: Operator set of the Horse Breeder. His goal is to s (Feed the horses) = 2, 0, 2 . The Horse Breeder is a SAS-PUC * 2 problem, solvable by Ponly.

	O	Pre	Post	Prevail v 19 , v 20 , v 21 , v 22 Explanation
	o 1 v 19

Table F .

 F 6: Operator set of the Innkeeper. He can work at the bar: s (Serve a drink) = 1, 2, s 0 [v 20 ], 1 ; and manage the Inn: s (Assign a room) = s 0 [v 18 ], s 0 [v 19 ], 2, 1 . The Innkeeper is a SAS-PUSC 0 problem, solvable by both Pand P.

	O	Pre	Post	Prevail v 10 , v 11 , v 12 , v 13 Explanation
	o 1 v 10

Table F .

 F 7: Operator set of the Native hunter. His goal is to s (Hunt) = 2, 0, 1, 1 ). The Native hunter is a SAS-PUSC 2 problem, solvable by both Pans P.

  Call Of Duty, The Witcher 1 & 2, Pacman...) à d'immenses mondes ouverts (The Elder Scrolls : Skyrim, Gran Theft Auto, Assassin's Creed, Red Dead Redemption,...), et même des openworlds infinis grâce à la génération procédurale (Minecraft, No Man's Sky...). Certaines histoires sont tellement immersives que vous avez l'impression d'être un acteur dans un film (The Witcher 3, Detroit : Become Human, Fable : The Lost Chapter,...). Les casques de réalité virtuelle sont proches de se retrouver dans tous les foyers, surtout avec l'annonce du Metaverse. Autant dire que les nouvelles technologies sont nombreuses pour créer des jeux vidéo toujours plus immersifs. Il est donc important de peupler ces environnements de haute qualité avec des PNJ crédibles, c'est-à-dire des PNJ qui ne se contentent pas de patrouiller ou d'interagir sans but avec le joueur. Les PNJ crédibles sont des PNJ qui vivent dans l'environnement, qui semblent avoir une vie réelle, quoi que fasse le joueur. Si les PNJ doivent intéragir avec le joueur, leurs actions doivent être cohérentes en fonction de leurs objectifs. C'est là que la planification est intéressante, en fonction de du but d'un PNJ, le planificateur va essayer de créer un plan d'action qui relie l'état actuel du PNJ à l'état but. Un plan d'action est plus pertinent qu'une seule action dans le sens où la première action du plan est utile pour une ou plusieurs actions qui viendront ensuite. Par exemple, un PNJ qui transporte une caisse peut le faire pour renforcer une position, et le même PNJ utilisera ensuite sa position renforcée pour se mettre à couvert afin de préparer une embuscade et ainsi de suite. Goal-Oriented Action Planning (GOAP) [7] est le planificateur le plus utilisé dans les jeux vidéo. F.E.A.R. [2], Shadow of Mordors [3], Assassin's Creed [5], Rise of the Tomb Raider [4], tous ces jeux ont utilisé GOAP pour rendre leurs PNJ plus crédibles. Les PNJ de F.E.A.R sont, à ce jour, toujours considérées comme faisant partie des meilleurs PNJ de l'industrie du jeu vidéo [8]. Les piétons, la faune, les ennemis des récents Assassin's Creed (Odyssey, Valhalla...) sont contrôlé via GOAP : il rend ces jeux plus dynamiques et immersifs pour les joueurs

  (Domain Operator Graph et Operator Graph) et représentent le problème de planification, le troisième est un graphe d'actions (Action Graph) et est une représentation graphique du plan solution. Ces graphes axés sur les opérateurs et les actions mettent en lumière les opérateurs et les actions au lieu des états, des valeurs d'état ou des variables d'état. De plus, la sortie d'un planificateur SAS étant des plans d'action, il est pratique de les représenter sous forme de graphe pour visualiser les relations entre les actions. Comme mentionné ci-dessus, il existe des opérateurs (PU) qui bouclent ensemble et, par conséquent, pour certaines instances de problèmes, ils peuvent être instanciés un nombre exponentiel de fois, créant ainsi des plans de taille exponentielle. Ces opérateurs bouclant entre eux, bouclent via leur dépendance post-pré. Ces dépendances sont observables grâce aux Domain Operator Graph. Je prouve que dans ce graphe, lorsqu'il existe, le cycle formé par certains opérateurs est unique. Ce qui me permet de définir: Definition G.4.3. Soit v i ∈ M, Cycle(v i ) est l'ensemble des opérateurs de l'unique cycle du Domain Operator Graph sur v i .

Only 10% of the processing budget is granted for AI in video games.

Except for the cost of EscapeDanger which is equal to 0.5.

Crystal Dynamics in Rise of the Tomb Raider have used dynamic costs[4] [19].

The difference between structures SAS, SAS * and SAS + are recalled in subsection 3.1.3

The operators of the Native Hunter are given in Table F.7, Appendix F.

The reader is invited to read C.Bäckström thesis, Chapter 3, for more details on contradictory values.

I formally define the operator identifiers in Definition 4.1.1.

result is a function defined by C.Bäckström in Definition 3.4 [6, p.58], it merely returns the state resulting from executing an action sequence.

v 0 , a 0 v 0 , a 1 v 0 , a 0 v 0 , a 1 v 0 , a 0 v 0is therefore a possible sequence of actions where both take and drop operators have been instantiated three time: ope(a1 v 0 ) = ope(a 1 v 0 ) = ope(a 1 v 0 ) = o 1 v 0 and ope(a 0 v 0 ) = ope(a 0 v 0 ) = ope(a 0 v 0 ) = o 0 v 0 .It is exactly the same behaviour with the operators of the 3-bits Gray Code (cf. Figure 4.1), the operators of the pairs o+ v 0 /o - v 0 , o + v 1 /o - v 1 or o + v 2 /o - v 2 loopbetween each other and the induced cycles enable o + v 0 and o - v 0 to be instantiated twice by the planner in order to solve = (Φ, s 0 = 0, 0, 0 , s = 1, 0, 0 ). Figure4.1 shows the operator graph of the Horse Breeder and that of the 3-bits Gray Code. The blue directed edges represent the post-pre dependencies (E pre (O)), whereas the green directed edges represent the post-prv dependencies (E prv (O)). It may exist cycles through the green edges (post-prv dependencies) or through the green and the blue edges. By definition of the pre-and post-conditions, it is through the post-pre dependencies that values of state variables are modified. I therefore focused on the cycles created by the blue arcs in order to spot what cause the minimal solution plans to grow exponentially in some problems. The following therefore introduces the Domain Operator Graph which is an operator graph that focuses on post-pre dependencies only. This is trough this type of graph that chains of actions are built.

I could have created a notation to distinguish the operator set with ghost operators (like O + ) with

A loop in a graph is when there is an edge from a vertex to itself.

n-2 occurrences of δv 0

It is possible to add parameters to the operators of RDR-2 and HZD as well, but they are useless while planning. That is why I did not spend time adding parameters in these benchmarks. The benchmark ACO is enough to understand how to contextualize actions convincingly without affecting the behavior of the planning system.

We used ⊕ as a concatenation symbol here.

FPS: Frames per Second

I have kept the state variable indices for the SAS-PUSC 2 version of RDR-2. That is why the Dancer still has a state variable domain on v 27 even if only 20 state variables remain.

The slope of HZD is lower than the one of the SAS-PUC * 2 version of RDR-2 but the curve is still above the one of RDR-2 in Figure5.16. This inaccuracy is linked to the noise of the data.

P was written with the same configuration.

It should be noticed that these two BuildChain, (3.17) and(3.21), build the minimal chain: σ

= δ vi (s 0 [v j ], s 0 [v j ]) = δ vi (s 0 [v j ], q) ⊕ δ vi (q, s 0 [v j ]).

Theorem 4.5.1. BuildChain satisfies its specification, so it is correct.

Proof. The proof is given Appendix D.

BuildChain is used by P to build chains of actions that respect the Υ 1 -Truth Criterion. I showed in subsection 4.3.2 that there are three cases to take into account with Υ 1 chains of actions between the start s 0 [v i ] and the goal s [v i ]:

We can highlight that:

is easy to understand in order to solve a problem instance. Indeed, a planner is designed to solve planning problem instances, therefore, for SAS-PU problem instances, finding all the chains of actions that solve s 0 [v i ] = s [v i ] is a relevant starting point. This is the output specification of the phase 1 of TopoPlan: find all the σ 1 , i.e. the minimal chains of actions that link the start to the goal states.

The action prevail-conditions then give instructions to the planner to correctly order the actions of the different chains together. This is done during the phase 2 of TopoPlan while respecting the Υ 1 -Truth Criterion. An action T that establishes one prevail-condition (Post-uniqueness) of another action A is an establisher (cf. Definition 3.2.5). If T is already ordered in a chain, then it is yellow and the Υ 1 -Truth Criterion tells us that: T must be ordered before A, and if N ext(T) exists, then it is a cloberrer and A must be promoted before N ext(T). If T is required but not yet ordered in a chain, then the problem instance is either not solvable, or the establisher is in a minimal chain of the form of σ 2 = δ v i (s 0 [v i ], s 0 [v i ]). In other words, if the problem instance is solvable, ope(T) is a requestable operator inside the cycle of its domain operator graph. Let

) is built during Phase 2, then the chain of actions on v i is of the form of σ 3 . The concatenation of σ 2 and σ 1 is made through the additional order created by BuildChain (cf. Appendix D). At the end of its procedure, BuildChain adds the order (b, F irst(σ 1 )) in the set E A , s.t. b ∈ N pre (F irst(σ 1 )). b ∈ N pre (F irst(σ 1 )) implies that: pre(σ 1 ) = post(b), and therefore post(b) = s 0 [v i ]. Then, post(σ 2 ) = s 0 [v i ], therefore by post-unicity and Υ 1 , Last(σ 2 ) = b. Eventually, (b, F irst(σ 1 )) = (Last(σ 2 ), F irst(σ 1 )), which corresponds to the concatenation: σ 2 ⊕ σ 1 . Eventually, if the input of P is a solvable problem instance of the SAS-PUC 0 , SAS-PUC S 2 or SAS-PUC * 2 , then Phase 2 returns a Υ 1 , non-linear, minimal solution plan that solves while respecting the Υ 1 -Truth Criterion. If the input is not a solvable problem instance, Phase 2 can yield a failure through the BuildChain procedure, otherwise it returns a sequence of actions that is not irreflexive.

The last phase is a topological sort performed by the procedure DFSTopo. This procedure is a depth first search topological sort, proven correct [40]. Its input specification The idea is to use the identification technique of Definition 4.1.1 so as to store operators in an hashing table, which can be done with an array (or a vector of vector in C++) for instance. Any action can then be instantiated in a constant time (O(1)) via the procedure FindPost by looking after the corresponding operator in the pre-built hashing table. It requires a preprocessing (l.xx to yy), however, but it allows C.Bäckström to reduce the worst-case time complexity of his algorithm from O(n 2 • m 2 ) to O(n • m 2 ).

Appendices

A

Modified Planning Algorithm for SAS + -PUS Problems

{If a is the first action to be inserted into F [i], then build ᾱv i and βv i .}

if β = and γv i = then

36:

Order Last( β ) before F irst( γv i ) in R

Order Last( ᾱv i ) before a in R

Such problem instances are equivalent to solving an instance of a SAS-PUC 0 problem. So the solution plan, if it exists, is necessarily Υ 1 .

In the following, I consider

, there is no meaningful minimal chain between + and +.

Therefore, chain v 1 (+, +) =

, the first and only meaningful chain is:

It results that the minimal chain of actions that could have created a minimal solution plan more than Υ 1 is always Υ 1 . Eventually, SAS-PUC S 2 problems with one cycle affected by C S 2 are Υ 1 .

Induction Hypothesis

1. Initialisation: Let consider P(1), we have:

N pre (a 0 v i ) is a singleton due to (P) and we have a ∈ N pre (a

0), and P(1) holds.

2. Maintenance: Let consider P(k) to be true, and, at the end of the k th loop, we have y = pre(a k-1 v i ) = k. So y = k at the beginning of the (k + 1) th loop. At the end of the (k + 1) th loop, we have:

) and it is minimal because a k v i exists (otherwise it is a ghost operator and the procedure would have failed (1.9)) and it was not in δ v i (k, 0) at the beginning of the (k + 1) th loop (otherwise, a k v i would have been yellow and the procedure would have failed (1.11)). The reasoning about the minimality of the chain is similar to the Proof 3.2.1. Therefore, the loop invariant holds at the end of each iteration of the while loop.

3. Termination: First of all, s, g ∈ D v i and D v i is a finite set of value, so if it exists a chain of actions from s to g, then the size of the chain is finite and the while loop terminates with the stop condition: pre(chain v i (s, g)) = s.

The while loop does not terminate iff the domain operator graph G O (v i ) is not weakly connected and s and g are the post-condition of two operators that do not belong to the same graph. I demonstrated in subsection 4.2.2 that in such cases, it cannot exist a chain of actions between s and g. BuildChain builds in a backward manner the chain of actions, thus starting with a g v i . So BuildChain navigates through the domain operator graph that contains ope(a g v i ). Lemma 4.2.2 implies that, the while loop either (1) ends up with a ghost action, or (2) ends up cycling through the unique cycle of the domain operator graph because it cannot reach s. the if condition (1.9) handles the case (1) while the if condition (1.11) handles the case (2). A verification of the weakly connected condition can be done during the Setup. If so, (1.9) and (1.11) are no longer required. In particular, Lemma 3.2.1 says that a minimal chain is Υ 1 , so if the minimal chain exists the while loop cannot fail with (1.11).

Eventually, the finite size of D v i and the two if conditions (1.9) and (1.11) ensure that the while loop will always terminate.

Correctness:

The input specification imposes that s and g are in D v i , so if the domain operator graph G O (v i ) is weakly connected, then it exists a chain from s to g. In particular, it exists a minimal chain from s to g. Lemma 3.2.1 implies the minimal chain is Υ 1 so the while loop will always terminate. Eventually, the exit condition and the loop invariant both hold so the procedure BuildChain meets its specification.

Corollary D.1.1. The procedure BuildChain is complete.

Proof. If there exists a chain of actions between s and g, then it exists a minimal chain of actions between s and g and it is unique (cf. Lemma 3.2.2). If BuildChain fails, (1.9) or (1.11), then there is no solution (cf. Termination of the proof D.1.1). So it contradicts the fact that there exists a chain of action. If it does not fail, Theorem D.1.1 implies BuildChain returns a correct solution, i.e. the unique minimal chain of actions between s and g.

Corollary D.1.2. The procedure buildChain yields a failure if an action instance belongs to at least two different minimal chains.

In other words, this corollary says that BuildChain yields a failure if the solution plan being built is Υ k with k ≥ 2.

Proof. Suppose that BuildChain has built the minimal chain δ 1 . For all a ∈ δ 1 , a is colored yellow. Suppose that BuildChain is building δ 2 and, at the k th loop, a is the next action to be added to δ 2 . Because a is yellow, BuildChain yields a failure with (1.11). Therefore, BuildChain prevents the creation of minimal chains having the same actions.

D.2 Proof of TopoPlan P

I will now prove the correctness of TopoPlan (P).

Definition D.2.1. The specification of TopoPlan is:

• Output: If is solvable, then P returns ∆ a linear, Υ 1 and minimal solution plan that solves while respecting the Υ 1 -Truth Criterion. If is not solvable, P yields a failure. Theorem D.2.1. The algorithm TopoPlan satisfies its specification, so it is correct.

Proof. TopoPlan works in three phases. From (3.2) to (3.6) is Phase 1, from (3.9) to (3.31) is Phase 2 and, finally, from (3.32) to (3.36) is Phase 3.

In Phase 1, there is a for loop

the loop invariant of (3.2) is:

In this phase, TopoPlan aims at building a minimal chain of actions for each state variable whose start is different from the goal. The building of these chains is made by BuildChain which is correct (cf. Theorem D.1.1). Each minimal chain is unique (cf. Lemma 3.2.2) and independent from one state variable to an other. It results that

) is also true by independence and because BuildChain is correct no matter the state variable. Concerning the termination, M is a finite set so, if BuildChain does not fail, the for loop (3.2) always terminates. If BuildChain fails, then it means a chain of actions between the start and the goal does not exist, so the problem instance is not solvable. If the problem instance is solvable, the loop invariant holds and the loop always terminates. So P (3.2) (|M|) holds. At the end of Phase 1, we therefore have:

∆ 1 is a non-linear action plan with:

And ∆ 1 is obviously minimal. Each action of A 1 are yellow due to BuildChain. The specification of Phase 2 is:

• Input: = (M, O, s 0 , s ) and:

• Output: ∆ 2 = (A, E A ) a non-linear sequence of actions that is Υ 1 where each action is correctly ordered according to the Υ 1 -Truth Criterion.

There are two for loops ((3.9) and (3.10)), so I described two loop invariants.

.l] is the set of actions: A[1.

.l] = {a 1 , ..., a l }. And so, A[1.

.|A|] = A.

has been established and declobbered with respect to the Υ 1 -Truth Criterion.

I starts by proving that the for loop (3.10) is correct. That is, whenever the loop invariant P (3.10) holds and the loop (3.10) terminates, then

) holds, then it means a p v i has been ordered with respect to the Υ 1 -Truth Criterion.

N prv (a

} is the set of actions that establish the defined prevail-conditions of a p v i . Due to (P), there is exactly one action per defined prevail-condition of a p v i , and therefore there is no two actions in N prv (a p v i ) that affect the same state variable. Therefore, at each loop, the establishment of a prevail-condition of a p v i is independent of the others. In other words, if

] by independence of each loop. Let a p v i be the action to assert and a q v j ∈ N prv (a p v i ) one of its establisher (by definition of N prv ), that is:

with T = a q v j and C = N ext(a q v j ) via the instructions between (3.11) and (3.18) for s 0 [v i ] = q, and between (3.19) and (3.26) for s [v i ] = q. If a q v j is yellow, then it is already inside A so P only needs to order a q v j before a p v i (3.17). If a q v j is not yellow, then it is white and therefore not in A. In other words, a q v j ∈ O \ A so P builds a chain of actions from s 0 [v j ] to q, with the procedure BuildChain (3.15) proven correct, in order to establish the prevail-condition of a p v i . The actions of this chain are thus colored yellow and added to A, a q v j included and it can now be ordered before a p v i (3.17) to respect the Υ 1 -Truth Criterion. If N ext(a q v j ) = ∅, then the successor of a q v j has not yet been defined, which means that it is white and not in A. Indeed, any action of A are added by BuildChain which, for any action a of the chain, assigns a as successor to b ∈ N pre (a). Therefore, an action chain from q to s 0 [v i ] is missing, hence BuildChain (3.21) 1 . At the end of the procedure BuildChain, N ext(a q v j ) is defined and according to the Υ 1 -Truth Criterion it is the clobberer, therefore a p v i must be promoted before N ext(a q v j ) (3.24). Due to the restriction (S4), no action can both clobber and establish a proposition, hence the if condition (3.23).

Then, if q = s [v i ], then the proposition q has no clobberer, so P does not need to promote a p v i . If q = s 0 [v i ], then q has two possible establishers: the start s 0 [v i ] or the action whose post-condition is

-FillBucketWithWater satisfies pre(FillHorseThrough), it is colored yellow and added to A.

-BuildChain ends,

A = {FillHorseFeeder, TakeHaystack, FillHorseTrough, FillBucketWithWater} Phase 2: For all actions of A (3.9), check all their prevail-conditions via the set N prv (3.10). (3.11) and (3.19) are false. DropBucket ∈ Cycle(v v 1 ) that is concerned by C * 2 , so (3.27) is true. (DropBucket, TakeHaystack) is added to E A (3.28) (post-prevail dependency, possible promotion order). DropBucket remains white.

• N prv (FillHorseTrough) = {PickUpBucket}.

PickUpBucket is white (3.14), so BuildChain builds backwards (3.15): * PickUpBucket establishes prv(FillHorseTrough), it is colored yellow and added to A.

(PickUpBucket, FillHorseTrough) (3.17, establishment order) is added to

N ext(PickUpBucket) = DropBucket is now defined so, (FillHorseTrough, DropBucket) is added to E A (3.24 promotion order).

• N prv (FillBucketWithWater) = {PickUpBucket}. 20), so (FillBucketWithWater, DropBucket) is added to E A (3.24 promotion order).

• N prv (PickUpBucket) = {DropHaystack}. • TakeHaystack is green.

• FillHorseTrough is green.

• FillBucketWithWater is green.

• PickUpBucket is green.

• DropBucket is green.

The solution plan is: ∆ = PickUpBucket, FillBucketWithWater, FillHorseTrough, DropBucket, TakeHaystack, FillHorseFeeder Resolution of 2 = (Φ HB , s 0 = 1, 0, 0 , s = 2, 0, 2 ) by P : Phase 1:

-FillHorseFeeder satisfies s [v 0 ], it is colored yellow and added to A.

(TakeHaystack, FillHorseFeeder) is added to

, BuildChain builds backwards (3.4):

, it is colored yellow and added to A.

(FillBucketWithWater, FillHorseTrough) is added to

-FillBucketWithWater satisfies pre(FillHorseThrough), it is colored yellow and added to A.

A = {FillHorseFeeder, FillHorseTrough, FillBucketWithWater} Phase 2: For all actions of A (3.9), check all their prevail-conditions via the set N prv (3.10).

• N prv (FillHorseFeeder) = ∅

PickUpBucket is white (3.14), so BuildChain builds backwards (3.15): * PickUpBucket establishes prv(FillHorseTrough), it is colored yellow and added to A.

(PickUpBucket, FillHorseTrough) (3.17, establishment order) is added to

N ext(PickUpBucket) = DropBucket is now defined so, (FillHorseTrough, DropBucket) is added to E A (3.24, promotion order).

• N prv (FillBucketWithWater) = {PickUpBucket}. • N prv (PickUpBucket) = {DropHaystack}.

(PickUpBucket, TakeHaystack) is added to E A (3.24, promotion order).

• N prv (DropBucket) = ∅.

• N prv (DropHaystack) = ∅.

• N prv (TakeHaystack) = {DropBucket}. DropBucket turns green (2.11) and is added to ∆. DFSTopo succeeds. TakeHaystack turns green (2.11) and is added to ∆. DFSTopo succeeds. FillHorseFeeder turns green (2.11) and is added to ∆. DFSTopo succeeds.

• TakeHaystack is green.

• FillHorseTrough is green.

• FillBucketWithWater is green.

• PickUpBucket is green.

• DropBucket is green. ayant une complexité temporelle linéaire capable de résoudre ces nouveaux problèmes. Enfin, j'ai réalisé plusieurs expériences en utilisant plusieurs benchmarks différents pour évaluer l'efficacité de mon algorithme et vérifier qu'il respecte bien la contrainte temps réel.

G.2 Goal-Oriented Action Planning

Introduction : La planification en temps réel est l'une des alternatives [14] pour modéliser et modifier le comportement des personnages non joueurs (PNJ) dans les jeux vidéo commerciaux. Goal-Oriented Action Planning (GOAP) a été introduit pour la première fois dans F.E.A.R. [2], un jeu de tir à la première personne (FPS) sorti en 2005. Lorsque le but d'un PNJ est mis à jour en fonction de l'état actuel du jeu, GOAP recherche, en arrière, un plan pour atteindre ce nouvel objectif : GOAP commence par chercher une action qui peut atteindre ce nouvel objectif dans l'ensemble des actions du PNJ ; les préconditions d'une telle action deviennent de nouveaux buts à atteindre, et ainsi de suite jusqu'à ce que l'état actuel du jeu puisse satisfaire les préconditions d'une action. Pendant la recherche, des préconditions contextuelles sont utilisées pour interroger l'état actuel du jeu et calculer diverses valeurs numériques (par exemple, les distances à esquiver). Dans F.E.A.R., les actions sont implémentées en tant que classes C++, et la recherche réutilise A* [15] du module de pathfinding du moteur de jeu ; la fonction heuristique utilisée dans GOAP évalue chaque noeud de recherche en additionnant les coûts des actions, puis calcule le nombre de buts restant à satisfaire. Les coûts des actions dans F.E.A.R. ne sont pas dynamiques : une fois conçus et implémentés dans le jeu, les coûts des actions sont des valeurs constantes qui ne seront jamais modifiées pendant la recherche.

Dans ce chapitre, j'aborde l'utilisation des coûts d'action dans GOAP, et plus généralement la représentation des actions en pratique dans les jeux commerciaux utilisant GOAP. En effet, ce système gère les situations de jeu inattendues mieux que d'autres alternatives d'ingénierie comportementale ; mais les concepteurs de jeux ressentent un manque de contrôle [14] sur une architecture d'IA capable de produire des comportements inattendus pour les PNJ [7]. Par conséquent, les coûts d'action apparaissent comme un moyen de modifier ce qui se passe à l'écran, ce qui entraîne la question suivante pour une nouvelle équipe de développement de GOAP : comment concevoir les coûts d'action ? L'IA de F.E.A.R. a été largement reconnue comme la meilleure de sa catégorie lors de sa sortie [16], et l'est toujours [8]. Aussi controversées que puissent être ces critiques, GOAP dans F.E.A.R. a néanmoins eu un grand impact parmi les développeurs d'IA de jeux et a définitivement suscité l'intérêt des planificateurs dans les jeux vidéo commerciaux en général et dans les FPS en particulier.

Grâce au SDK publié par Monolith en 2006 [10], la difficulté de développement de l'IA des jeux n'est plus un problème de codage : le code C++ de GOAP de F.E.A.R. peut être étudié et ses principales parties peuvent être dupliquées. Bien qu'aujourd'hui de nombreux jeux utilisent GOAP pour le comportement des PNJ, F.E.A.R. reste à ce jour le seul jeu dont le SDK permet d'accéder au code C++ du planificateur. Il existe également plus de 300 dépôts GOAP sur Github, implémentant des fonctionnalités de planification pour de Proof. cf. manuscrit Soit Cycle(v i ) = {a + , a -}, L'idée de la classe SAS-PUC * 2 est que, pour chaque instance de ces problèmes, les actions sollicitant l'instance de o + peuvent être ordonnée indépendamment des actions sollicitant l'instance de o -. Ainsi, le planificateur peut instancier o + et ordonner toutes les actions sollicitant a + , puis instancier o -et ordonner toutes les actions sollicitant a -, ou inversement.

Maintenant que mes nouvelles classes de problèmes abordables et Υ 1 sont introduites, je vais présenter mon algorithme TopoPlan.

Mon Algorithme TopoPlan (P)

Cette section dans mon manuscrit présente l'algorithme TopoPlan, noté P, qui résout les problèmes SAS-PUC k qui sont au plus Υ 1 . Chaque instance de problème est notée = (Φ, s 0 , s ). P est basé sur P : il planifie en arrière, pour profiter de la post-unicité, et le fait en trois phases. La phase 1 construit les chaînes d'actions minimales entre le début s 0 et le but s pour chaque variable d'état v i ∈ M. Toutes les actions de ces chaînes sont colorées en jaune ; la phase 2 vérifie les prevail-conditions de toutes les actions jaunes. Elle crée des ordres entre les actions jaunes des différentes chaînes afin d'établir et/ou de promouvoir correctement chaque prevail-condition définie de chacune des actions jaunes. Soit v i ∈ M et p ∈ R O v i une valeur sollicitable, deux chaînes d'actions minimales, une entre s 0 [v i ] et p, et une entre p et s 0 [v i ] peuvent être construites pour assurer l'établissement de p tout en respectant le Υ 1 -Truth Criterion (cf. mon manuscrit). Finalement, un plan non linéaire minimal et Υ 1 est retourné ; la phase 3 trie topologiquement le plan non linéaire. P diffère cependant de P de plusieurs façons : j'ai ajouté des conditions supplémentaires pour prendre en compte la relaxation de la restriction (S) ; je suis allé plus loin dans le pre-processing et j'ai créé des ensembles supplémentaires d'opérateurs (cf. predecessor sets dans mon manuscrit) pour réduire la complexité de P ; Le pre-processing de P n'est pas effectué pendant la planification par rapport à P, mais pendant la compilation ; Enfin, P n'instancie pas les actions, il colore les opérateurs et renvoie des plans d'identifiants.

Dans cette section de mon manuscrit, je présente mon algorithme puis prouve sa complexité temporelle et spatiale.

G.5 Benchmarks et Résultats

Ce chapitre est divisé en deux parties suivies d'une discussion. La première section présente les benchmars évolutifs qui sont des benchmarks conçus pour croître par rapport au nombre de variables d'état (m) ou par rapport au nombre maximum de valeurs dans un domaine de variables d'état (n). Parmi ces benchmarks figurent les exemples BlockWorld et Tunnel [6]. J'utilise ces benchmarks pour renforcer empiriquement mes résultats théoriques. Le but de la deuxième section est de montrer que des benchmarks réalistes peuvent être représentés avec mes nouvelles classes de problèmes abordables. Parmi ces benchmarks réalistes figurent RDR-2, ACO et HZD qui sont respectivement basés sur les jeux vidéo commerciaux Red Dead Redemption [11], Assassin's Creed : Origins [12] et Horizon Zero Dawn [13]. J'utilise ensuite ces benchmarks pour comparer mon implémentation C++ de P et P. Les résultats sont impressionnants : P est environ > 34.000 fois plus rapide que P pour résoudre les instances de ces problèmes réalistes. Je termine le chapitre par une discussion qui explique pourquoi P est beaucoup plus rapide que P ainsi que d'autres différences intéressantes entre les deux. Je donne aussi des exemples et explications pour représenter des problèmes réalistes avec mes nouvelles classes de problèmes abordables. Voici quelques points en question :

• Bien que la relaxation de la restriction syntaxique (S) avec la restriction structurelle (C * 2 ) donne plus de liberté à la représentation des opérateurs, il y a encore des concessions à faire. Par exemple, il n'est pas possible d'avoir la situation suivante : considérons deux commutateurs On/Off où les deux opérateurs On sont requis par un opérateur A et les deux avoir d'autres explications à cette énorme différence de performance. Mon intuition est que, lorsque m est petit, le problème est suffisamment petit pour que le processeur AMD mette en cache toutes les fonctions P et le problème dans le cache L1.

G.6 Conclusion

Comment Réduire la Compléxité Temporelle pour un Passage à l'Échelle en Temps-Réel ? Goal-Oriented Action Planning J'ai mené une étude sur le planificateur GOAP de F.E.A.R. et j'ai découvert que les coûts des actions, qui étaient utilisés par la fonction de coût pour évaluer la cohérence d'un plan, sont en pratique éclipsés par les context preconditions. A tel point que les actions deviennent contextuellement post-uniques (1). Ensuite, je souligne, à l'aide du code source de F.E.A.R., que ses développeurs ont essayé de garder la représentation des actions simple, c'est-à-dire que chaque action a seulement quelques préconditions et une seule post-condition. Ce qui signifie que les actions sont unaires (2). Enfin, j'ai souligné que les plans d'action sont totalement ordonnés (3) et qu'il n'y a pas deux fois la même action dans ceux-ci (4). J'ai enfin transformé ces quatre points en hypothèses que j'ai utilisées tout au long de ma thèse.

Simplified Action Structure Je rappelle dans ce chapitre la théorie derrière le formalisme Simplified Action Strucute(SAS). Même si GOAP est basé sur STRIPS, il est en fait plus proche du formalisme SAS. De plus, (1) et (2), deux de mes hypothèses, sont des restrictions que C. Bäckström a utilisées pour créer des problèmes SAS + -PUS, qui sont des problèmes qui peuvent être résolus par un algorithme en temps polynomial. Comme mon objectif était de réduire la complexité de la planification en temps réel, j'ai donc implémenté l'algorithme SAS + -PUS en C++, appelé P, et je l'ai essayé sur des benchmarks réalistes, basés sur des jeux vidéo, qui forment des problèmes SAS + -PUS. Les résultats étaient encourageants car P est capable de planifier pour des centaines de PNJ en temps réel et en leur fournissant des plans d'actions plus longs que ce que les planificateurs actuels de GOAP sont capable de faire. La restriction (S) est cependant très restrictive.

Topological Planning Dans ce chapitre, j'ai relâché (S) et présenté de nouvelles restrictions structurelles à combiner avec (P) et (U) de manière à créer des problèmes abordables pour les jeux vidéo. Comme l'une de mes hypothèses est qu'il n'y a pas deux fois la même action dans un plan de solution, j'ai introduit une restriction de sortie dénotée Υ 1 et que j'utilise pour créer le Υ 1 -Truth Criterion, un critère de vérité qui, s'il est respecté, assure que le plan de solution est correct. Cependant, Υ 1 ne fournit pas d'instructions permettant de produire des problèmes abordables. J'introduis donc une nouvelle restriction, notée C k , dont le rôle est de restreindre à au plus k le nombre d'opérateurs sollicitables à l'intérieur du cycle unique de chaque Domain Operator Graph. En effet, j'ai montré dans Problèmes SAS-PUC k qui sont Υ 2 L'étape suivante après les problèmes SAS-PUC k qui sont Υ 1 est évidemment ceux qui sont Υ 2 , puis Υ 3 et ainsi de suite. À mon avis, l'étude du cycle unique dans les Domain Operator Graph reste cruciale pour ceux qui veulent explorer ces problèmes.

Une autre étape consiste à considérer les structures SAS * ou SAS + avec les restrictions (P), (U) et C k . Ces deux structures permettent aux variables d'état d'être indéfinies dans l'état but, et SAS + permet également à l'état de départ d'être indéfini. Cela ajoute une notion d'incertitude qui peut être intéressante pour certaines recherches.

Planification topologique hiérarchique Pour le Native Hunter, j'ai expliqué pourquoi j'ai fusionné les paires d'opérateurs EquipBow/UnequipBow et EquipQuiver/UnequipQuiver dans la paire GetReadyToHunt/Unequip. Les deux premières paires d'opérateurs ne respectaient pas la restriction C * 2 , alors que la fusion des deux oui. Mais, GetReadyToHung et Unequip peuvent en fait être vus comme des sous-buts dont les plans de solution minimale sont respectivement EquipBow, EquipQuiver et UnequipBow, UnequipQuiver . Cette observation ouvre la voie à la planification hiérarchique avec P. On peut créer des problèmes de sous-problèmes de sous-sous-problèmes... et ensuite appliquer P pour résoudre chaque instance de problème l'une après l'autre afin d'obtenir un plan de solution minimal pour un agent donné.

Appliquer TopoPlan dans les systèmes GOAP C'est une question de maturité technologique (TRL). J'ai prouvé théoriquement et empiriquement mon algorithme et je l'ai appliqué sur des benchmarks réalistes mais de manière abstraite. Les résultats sont impressionnant mais il faut maintenant tester P sur des mondes virtuels plus concrets, réaliste. L'intégrer dans un projet tout en s'assurant qu'il fonctionne et interagit bien avec les autres sous-systèmes est certainement une pertinente prochaine étape.