Guillaume Sarthou

Guilhem Buisan

Aurélie Clodic

Rachid Alami

Extending Referring Expression Generation through shared knowledge about past Human-Robot collaborative activity

Being able to refer to an object, a person, or a place in a non-ambiguous manner is a need when one has to achieve collaborative activities with a partner. This is the socalled Referring Expression Generation (REG) problem. While widely used for Human-Robot Interaction, state of the art approaches restrict its use to the current environment. We propose a novel extension to the REG which takes full advantage of the Human-Robot shared knowledge about past actions as additional information to generate Referring Expressions. We show that our approach is usable with a domain-independent ontology as a knowledge base and that it can also use a semantic representation of past activity to generate RE. We illustrate our method through simulated situations and discuss its efficiency and pertinence.

I. INTRODUCTION

When two or more agents perform a collaborative task in a shared environment, although they may have a different perception of this environment, they are able to estimate the information they share and can thus use it to communicate about some known entities. This assumption is commonly used to develop and evaluate Referring Expression Generation (REG) methods through the use of snapshots of the environment [START_REF] Duboue | Evaluating robustness of referring expression generation algorithms[END_REF]. It has also been used when the REG has been applied to Human-Robot Interaction (HRI). However, the joint activity performed during a collaborative task can be seen as additional knowledge shared by the involved agents. During the interaction, the agents perform actions on entities in the environment. We can thus refer to the entities through these past actions in addition to their properties.

Consider the situation represented in Fig. 1b as the current situation where the robot has to ask the human for the knife 2. Since the robot is performing another action of the joint task, it cannot see what is in front of the human and so it can not know any spatial relations about it. Therefore, the robot can only use knife 2 attributes (e.g. its color) to generate an expression referring to it. However, there are two other blue knives in the kitchen (1 and 3). Even if the one attached to the wall (1) can be considered as out of context as being already reachable by the robot and not by the human. The other knife (3) still leads to ambiguity and has no other attribute that differs from the one the robot has to refer to.

Until now, we only have considered the current situation (i.e. b) and not the human-robot shared experience. By taking into account a previous human action (Fig. 1a) which was to cut a tomato, the robot can find a way to refer to knife 2. A possible RE would be "the knife with which you cut the tomato". The exploitation of shared knowledge about past 1 LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France firstname.surname@laas.fr Referring to knife 2 in the current situation (b) is impossible if the robot is performing an action that does not allow it to see what is in front of the human. Considering a previous step of the human's task, the robot can refer to the knife through the action to cut a tomato (a). activity in addition to the usual properties for REG allows the generation of richer RE that could be easier to understand by the human partner. Besides, it allows to extend REs use to contexts where the previous method was not effective.

As explained in [START_REF] Gatt | Survey of the state of the art in natural language generation: Core tasks, applications and evaluation[END_REF], the REG task is composed of two parts. The content determination focuses on finding the set of relations to use. The linguistic realization consists in choosing the right words to communicate the content. This paper only focuses on the content determination. However, since the knowledge base we consider is not dedicated to this specific task, the method we present still ensures that the concepts used in the content determination (e.g. the actions or the object types) have a name in natural language without performing the linguistic realization.

This paper is an extension of our previous work [START_REF] Buisan | Efficient, situated and ontology based referring expression generation for human-robot collaboration[END_REF] proposing an algorithm to generate RE with an ontology as a knowledge base. Moreover, this REG algorithm has been integrated within a cost-based Human-Aware Task Planner to estimate the feasibility and cost of REs during the planning process [START_REF] Buisan | Human aware task planning using verbal communication feasibility and costs[END_REF]. The main contribution of this paper is an extension of the ontology-based algorithm for the generation of REs by considering past agents' activities. A side contribution of this paper is a formalism to represent Hierarchical Execution Traces (executed HTN-based plans) in an ontology. While our previous contribution considered only cost functions on properties and relations to represent the cognitive load required for a human to interpret the RE, we propose here to add customizable cost functions, based on time, to represent the cognitive load required for a human to remember referred actions and tasks.

In §II, we discuss related work on REG and HTN rep-resentation in ontologies, and how our method addresses new challenges. The knowledge bases, their update, and the representation of Shared Hierarchical Execution Trace (HET) in an ontology are presented in §III. The proposed algorithm is described in §IV. Comparison with our previous work is presented in §V to show the performance impact of the addition of the new feature. Finally, in the last section, we present and discuss five different cases to illustrate the solutions found by our algorithm depending on the agent's knowledge about past activities.

II. RELATED WORK A. Referring Expression Generation

Referring Expression Generation (REG) has been formalized as the concern of "how we produce a description of an entity that enables the hearer to identify that entity in a given context" [START_REF] Reiter | Building natural language generation systems[END_REF]. The criteria for a good RE have been refined over time through multiple algorithm propositions. In [START_REF] Dale | Cooking up referring expressions[END_REF] a Depth First Search (DFS) approach was proposed leading to non-optimal solutions with redundant information. The Full Brevity approach [START_REF]Generating referring expressions: Constructing descriptions in a domain of objects and processes[END_REF] solved this first issue to fit the Grice's Maxim of Quantity [START_REF] Grice | Logic and conversation[END_REF] but at the cost of an exhaustive search. With the Incremental Algorithm (IA) [START_REF] Dale | Computational interpretations of the Gricean maxims in the generation of referring expressions[END_REF] Dale introduced the notion of preference over features to represent the fact that some entities' features, such as the color, are easier to understand than others.

During the following decade, the Graph-Based Algorithm (GBA) REG [START_REF] Krahmer | Graph-based generation of referring expressions[END_REF], based on a branch&bound algorithm, introduced a labeled directed multi-graph representation known as the REG graph. Keeping the notion of preference over features through the assignation of costs to each edge of the graph, the REG graph allows the use of relations between entities in an easy way. The more notable extensions are the Longest First (LF) algorithm [START_REF] Viethen | Graphs and spatial relations in the generation of referring expressions[END_REF] trying to over-specify the description and [START_REF] Krahmer | Computational generation of referring expressions: A survey[END_REF] using the hierarchy of types.

Other approaches based on machine learning have been developed such as [START_REF] Yamakata | Belief Network based Disambiguation of Object Reference in Spoken Dialogue System[END_REF] using belief networks or [START_REF] Fitzgerald | Learning distributions over logical forms for referring expression generation[END_REF] using a log-linear model trained from a corpus of the probability distribution of REs. However, these kinds of approaches suffer from a lack of genericity by being restricted to the learned relations.

Two major approaches have been used in robotics architectures. In one hand the DIST-PIA [START_REF] Williams | Referring expression generation under uncertainty: Algorithm and evaluation framework[END_REF], a distributed Incremental Algorithm, integrated in a robotic architecture in [START_REF] Williams | Dempster-Shafer theoretic resolution of referential ambiguity[END_REF]. It uses consultants on each of the distributed knowledge bases of the architecture and has an IA querying these consultants. In the other hand, an ontology-based REG has been presented in [START_REF] Ros | Solving ambiguities with perspective taking[END_REF]. However, while ontologies can be viewed as a more complete REG graph, this method is not able to use relations to other entities such as in the original GBA REG. This issue has been solved by our work [START_REF] Buisan | Efficient, situated and ontology based referring expression generation for human-robot collaboration[END_REF] in which we proposed an ontology-based REG using a Uniform Cost Search algorithm.

In all the previous works, the REG is only performed on the current environment state. [START_REF] Williams | Toward forgetting-sensitive referring expression generationfor integrated robot architectures[END_REF] is the first to add a temporal aspect by considering a sequence of REG and trying to re-use properties from previous descriptions along with a forgetting model based on decay or interfering. This method has been tested on a "Guess Who"-style game implying that the used properties hold between the REG and thus can be re-used. However, this assumption can no longer be maintained in a real dynamic interaction where objects are manipulated. [START_REF] Oberlander | Generating expressions referring to eventualities[END_REF] showed that generating references to eventualities (i.e. past actions) can be done in the same way as generating references to physical entities but does not generate references to entities through the use of past actions. Finally, [START_REF] Wiriyathammabhum | Referring to objects in videos using spatio-temporal identifying descriptions[END_REF] uses RE involving past actions to identify the referent in videos but does not generate them.

The determination of the properties' costs will not be discussed here but we can mention [START_REF] Belke | Tracking the time course of multidimensional stimulus discrimination: Analyses of viewing patterns and processing times during "same"-"different "decisions[END_REF] and [START_REF] Koolen | Learning preferences for referring expression generation: Effects of domain, language and algorithm[END_REF] which use learning techniques to estimate the users' preferences.

B. HTN-based tasks representation in ontology

In HRI, storing and reasoning about past activities is needed both to learn from experience [START_REF] Petit | Reasoning based on consolidated real world experience acquired by a humanoid robot[END_REF] and to speak about what happened [START_REF] Mealier | Narrative constructions for the organization of self experience: Proof of concept via embodied robotics[END_REF]. Some approaches represent the past actions using structured sets of SQL tables [START_REF] Mealier | Narrative constructions for the organization of self experience: Proof of concept via embodied robotics[END_REF], but such a representation lacks of semantic information both on the involved entities (e.g. a robotic agent is a specific type of agent) and on the actions (e.g. a cut action is part of a salad preparation task). Since ontologies are fully suitable to represent semantic information about entities and their relations, they have been used to represent task planning knowledge. In [START_REF] Sun | Rtpo: a domain knowledge base for robot task planning[END_REF], a Robot Task Planning Ontology (RTPO) is proposed but the model does not consider the rich semantics involved by the hierarchical nature and intricacies of human-robot joint activities.

HTN is a very popular way for representing, planning, and controlling autonomous agents' activities [START_REF] Ghallab | Automated planning -theory and practice[END_REF], [START_REF] Ingrand | Deliberation for autonomous robots: A survey[END_REF]. They are widely used for robotic planning as they allow to efficiently find complex plans by choosing between different task decompositions depending on the world state. In HRI scenarios, their usefulness is even more apparent. In [START_REF] Lallement | HATP: An HTN Planner for Robotics[END_REF], an HTN is used to generate human-robot joint plans. Furthermore, the hierarchical structure can be used to negotiate or communicate high-level plans when details do not matter [START_REF] Milliez | Using human knowledge awareness to adapt collaborative plan generation, explanation and monitoring[END_REF].

To the best of our knowledge, few works exist on the representation of an HTN and their Hierarchical Execution Trace (HET) in ontologies. [START_REF] Umbrico | An ontology for humanrobot collaboration[END_REF] describes the notion of complex tasks composed of simple tasks but does not go further in the representation. In [START_REF] Freitas | Using ontologies as semantic representations of hierarchical task network planning domains[END_REF] only the planning domain is represented. The major issue is that the ontology classes are used to describe the general HTN concepts (i.e. action, method) while the field concepts (e.g. the cut task) are described using the ontology individuals. Hence, this representation does not allow to represent sub-tasks and actions instantiations leaving a gap between the domain as a high-level knowledge and the HET as an instance of it. Our work is closer to BOWL [START_REF] Ko | Business-owl (bowl)-a hierarchical task network ontology for dynamic business process decomposition and formulation[END_REF], an HTN ontology for business process representation. Even if they have defined some specific relations for the Business-to-Business field, the general tasks, decomposition, and tasks links representation are interesting. However, BOWL only represents the HTN and not HETs, but does not preclude it.

III. STRUCTURING AND GATHERING THE KNOWLEDGE

In this section, we present first the main knowledge structure necessary to perform the extended REG through shared knowledge about past Human-Robot collaborative activity. We continue with an overview of the robotic architecture allowing to acquire these knowledge and we end this section with the semantic knowledge base internal structuring that we propose to represent the HTN and the HETs.

A. The three knowledge representations

Three knowledge representations are used and can be grouped into two categories: the dynamic and static ones. The dynamic part is updated all along the course of action, it is defined as K = K S , K E with K S the semantic part and K E the episodic one. The static knowledge base represents the planning domain as a Hierarchical Task Network (HTN).

Since this work addresses HRI applications, we consider that the robot maintains a semantic and episodic knowledge base per human it is interacting with (K Hi S and K Hi E ) in addition to its own (K R S and K R E ). This consideration allows to implement theory of mind decisional mechanisms [START_REF] Baron-Cohen | Does the autistic child have a "theory of mind[END_REF]. While the robot's own knowledge base is its personal truth, the agents' knowledge bases represent its estimation of the knowledge of its human partners.

1) The Hierarchical Task Network: An HTN is a way of representing a task to be planned, i.e. to be fully refined and instantiated. The following definitions are based on [START_REF] Erol | Htn planning: Complexity and expressivity[END_REF]. It is defined as a set of tasks N . A task n is represented as a task name with a list of arguments. For example, in Fig. 2, the task cut has the arguments A, V, K, which are respectively an agent, a vegetable, and a knife.

A task n can either be a primitive task (n ∈ P t) or an abstract task (n ∈ At), where P t is the set of primitive tasks (which do not need any refinement) and At the set of abstract tasks (needing further decomposition). In Fig. 2, PrepareSalad() ∈ At while cut(A,V,K) ∈ P t.

Then, we define the set of decompositions D. A decomposition is a pair (υ, N ) ∈ D where υ ∈ At is an abstract task and N a task network describing one way of decomposing υ. In Fig. 2, the abstract task PrepareVegetable(A,K,V) has two decompositions d8 and d9.

Given an initial state and an initial task network, a planner (such as HATP [START_REF] Lallement | HATP: An HTN Planner for Robotics[END_REF]) elaborates a plan. A plan is a sequence of primitive tasks such as they result from successive decompositions of the initial task network while respecting the constraints issued by the initial state. The planner thus tries to recursively select a decomposition for each abstract task in the task network until it reaches primitive tasks. Besides, the planner has to ground every argument of the tasks into entities of A (the Abox of the knowledge base K S ) while respecting constraints on them (e.g. types of arguments). The domain of the high-level task PrepareMeal. It is used by the planner (HATP) to elaborate an HR shared plan through a contextbased decomposition and parameter instantiation process (which vegetable V, which knife K, ...) including the selection of which agent A (Alex the human, or PR2 the robot) subtasks and/or actions will be allocated.

2) The semantic knowledge base -K S : Keeping the definition used in [START_REF] Buisan | Efficient, situated and ontology based referring expression generation for human-robot collaboration[END_REF], the semantic knowledge base K S is defined by K S = A, T, R with A, T and R being respectively an ABox, TBox and RBox. The detail of the structure of the semantic knowledge base is detailed in [START_REF] Buisan | Efficient, situated and ontology based referring expression generation for human-robot collaboration[END_REF]. The list of symbols used is presented bellow. We store in the ontology both the facts representing the current state of the world (e.g. the knife is on the table) and the descriptions of the executed tasks, as presented in §III-C.

3) The episodic knowledge base -K E : The episodic knowledge base K E is a timeline defined as

K E = { α, T }.
It is a set of pairs with α an instance of a task and T a temporal interval composed of two numerical values. The tasks defined in K E are also represented as entities of K S (α ∈ A). Since they are semantically represented in K S we can, inter alia, retrieve their types or arguments.

B. The knowledge gathering scheme

In the previous subsection, we saw the three knowledge bases. We now describe how they are interconnected and how the semantic and episodic ones are updated. The minimal robotic architecture we use is represented in Fig. 3.

The biggest box at the center represents the ontology component which maintains the semantic and episodic knowledge bases: an estimated knowledge base per each human (K Hi S and K Hi E ) in addition to the robot own knowledge base (K R S and K R E ). First of all, the HTN-based task decomposition description is parsed and stored offline (dotted arrow on Fig. 3) to store its abstract and primitive tasks as well as their parameters and their hierarchical links as concepts and properties of the ontology (see next section for details). The other arrows represent the online interactions between components. The situation assessment continuously updates the semantic knowledge base of each agent, which themselves temporally stamp these data in their respective episodic knowledge base. On request, the HTN planner generates a plan for the supervision [START_REF] Baron-Cohen | Does the autistic child have a "theory of mind[END_REF]. During plan execution, the supervision describes the performed tasks semantically (detailed in the following subsection) (2a) and stamps them in the timeline (2b) with their respective begin and end times. At task execution, the supervision also gathers data from the episodic knowledge base of the human partner (2c), to monitor their tasks. Upon a REG request from the supervision, the REG component explores the listener's knowledge bases (3) and returns the generated RE [START_REF] Buisan | Efficient, situated and ontology based referring expression generation for human-robot collaboration[END_REF].

When the supervision component inserts the executed tasks in the semantic knowledge base, it creates a hierarchical execution trace (HET). The execution trace can differ from the initial plan since it can be the result of plan repair or replanning steps within the same global task achievement. The HET thus contains the actions which have been performed and their link to the higher level of abstraction. No forgetting mechanism has been implemented yet since we focus on "short" interactions (few hours) but adding one would avoid the knowledge bases to grow indefinitely and also represent the human forgetting mechanism. This is for future work.

C. Building the ontology

The aim of the following representation is not for planning per se (since this is done by the human-aware task planner) but rather to allow to store and manipulate a description of the execution of the human-robot shared plans together with their hierarchical structure and the information provided by the situation assessment component. What is provided is the hierarchical task decomposition together with a semantic description of the entities used as tasks and action parameters, their properties, and relations to other entities in Fig. 3.

A "minimal" robotic architecture allowing to acquire and store the knowledge necessary to perform a REG using the past HR activities. The dotted arrow represents an offline acquisition. The other arrows are online interactions between the components. The numbering represents the execution order during the execution of a task, as explained in §III-B. The robot knowledge bases (K R S and K R E ) and the estimated mental states of its two human partners (K Hi S and K Hi E ) are updated permanently by the Situation Assessment component which tracks changes in the environment and by the robot supervisor which controls robot planning and execution activities and monitors humans actions.

the environment. Moreover, the descriptions presented below are automatically generated from an HATP domain and plan description [START_REF] Lallement | HATP: An HTN Planner for Robotics[END_REF].

1) HTN in ontology: An HTN represents the general knowledge about how to decompose high-level abstract tasks into executable primitive tasks. Since it is a piece of general knowledge, the domain is represented in the TBox and the RBox of the ontology.

We first define the classes and relations common to any HTN representation. The upper class in the TBox is HtnConcept from which inherit the classes HtnDecomposition and HtnTask. The HtnTask class is then refined into HtnAbstractTask and HtnPrimitiveTask. The RBox is composed without hierarchy of the properties hasDecomposition, hasSubtask, hasParameter, and their inverse (e.g. (hasP arameter, isP arameterOf ) ∈ Inv). The property hasDecomposition links an abstract task to its decompositions and the property hasSubtask a decomposition to its sub-tasks (primitive or abstract).

Thus, to represent an HTN P l = P t, At, D we add to our ontology a new class t for each : (1) primitive task n such as

n ∈ P t ⇔ t ∈ T ∧ (t, HtnPrimitiveTask) ∈ H (2) abstract task υ such as υ ∈ At ⇔ t ∈ T ∧ (t, HtnAbstractTask) ∈ H (3) decomposition d such as d ∈ D ⇔ t ∈ T ∧ (t, HtnDecomposition) ∈ H.
We then add to our ontology new relations r for each decomposition d = (υ, N ) such as: Then, the PrepareVegetableDecomp class, inheriting from the HtnDecomposition concept, is the upper class of all the PrepareVegetable task decompositions. Representing the first of these decomposition, PVDecomp A, we state that it has two subtasks (hasSubtask) Peel PVDecomp A and Cut PVDecomp A. The sub-tasks are specified into the decomposition they are performed in order to keep track of their context. The tasks' parameters are represented using the hasParameter property. Unlike the previous properties, this one is not directly applied to the classes but rather refined to be used by the task instances. Indeed, this property is refined into several ones, being hasParameter.i with i ∈ N (i.e. hasParameter.0, hasParamter.1, ...) aiming at representing the parameters by their position in the task parameters list. These refinements are independent of the translated HTN. Each of these properties is then refined and specified for every task of the current HTN. This second specification aims at representing the parameters by their name in the task parameters list, and link them with their position (via their inheritance link). Taking the example of the primitive task Cut of Fig. 2, the generated description is presented in Desc. 2 for the two first parameters. The parameter A is at position 0, we thus define the property Cut hasParameter.A that is a refinement of the property hasParameter.0. The task parameter is then set as the property domain, and the type of the argument (defined in the HTN) is set as the property range. For example. the property Cut hasP arameter.A has for domain the Cut task class and for range the Agent class, representing the agent performing the task. The same process is performed for every parameter of each task. 2) HET in ontology: The HTN planner (here HATP) first generates a hierarchical plan for a given human-robot collaborative task. This plan is then executed by the robot and its human partner. Whenever a task (abstract or primitive) is executed or its execution is perceived by an agent, its description is inserted in the agent's KB. The perceived tasks are thus instances of tasks and have to be grounded to the HTN ontology described above. Let us take the example of an agent perceiving an instance of a decomposition during the execution of a plan. We show in Desc. 3 a partial representation of this instance in our ontology. Decomp PV 3 (part of the plan) is of type PVDecomp A (in the domain) the first decomposition of the abstract task PrepareVegetable, described in Desc. 1. Through the property DecompositionUsedBy we state that this decomposition comes from the instance PV 3 of the abstract task PrepareVegetable. With the property hasSubtask, we then describe that it is composed of two instantiated sub-tasks Peel 5 and Cut 7. The primitive task Cut 7 is described as being an instance of Cut PVDecomp A (the task Cut coming from the first decomposition of the abstract task PrepareVegetable). The task parameters are also instantiated with respect to the specifications of the hasParameter properties. Here we see that the parameter A, representing the agent, is grounded as Alex. By doing so, we successfully represent the semantic of any executed task together with its parameters and the link with higher level of plan execution description. Description of the initiation of a decomposition of a PrepareVegetable task and a primitive Cut task resulting from a plan and linked to the description of the domain.

(1) d ∈ D ⇔ (υ, hasDecomposition, d) ∈ E (2) n ∈ N ⇔ (d, hasSubtask, n) ∈ E.

IV. ALGORITHM

On the basis of the newly introduced knowledge available to the system (i.e. the agents' past actions), in this section, we present an extension of the Uniform Cost Search (UCS) REG algorithm that we introduced in [START_REF] Buisan | Efficient, situated and ontology based referring expression generation for human-robot collaboration[END_REF]. We first briefly recall the key aspects of the REG problem and of our algorithm and then, we present the modifications which have been made to deal with the agents' past actions (the HET).

A. Problem definition

In its simplest form, the REG problem is a pair P = a t , K where a t ∈ A ⊂ K is an entity we are aiming to designate in an unambiguous way through its relation to other entities. A solution to the REG problem is a set of relations, in the form of triplets, which could be verbalized. In the case of an entity with no label (i.e. an anonymous entity), this entity cannot be verbalized and thus cannot be referred to directly. For example, in the simple sentence "the cucumber" where the entity to refer to has no name. We thus refer to it through its type Cucumber which has a verbalizable label. In the triplets of the solution, these anonymous entities are prefixed with a question mark (e.g. ?c isA Cucumber) and represented as variables in the set of variables X.

To be valid, a RE E must satisfy three constraints. C1) Nameability of entities: each entity a ∈ A used in any relation of E must have a label (L a (a) = ⊥). C2) Nameability of the variables; each anonymous entity (variable) x ∈ X, used in any relation of E, must be involved in a typing relation ((x, "isA", t) ∈ E with t ∈ T in a way that the entity's type has a label (L t (t) = ⊥). C3) Correct instantiation of variables; for each variable x ∈ X used in E, there must exist a substitution function f : X → A. This means that there must exist at least one instantiation of each variable for which all the relations of E must exist in K S .

To be a solution, a reference must be valid and the variable x t representing the target entity must have only one possible substitution that is a t .

B. Algorithm extension

Our original REG algorithm is based on an Uniform Cost Search algorithm. It constructs bigger and bigger RE candidates until a valid one is found. A node in the search tree is composed of a cost and a candidate RE, i.e. a set of relations T ⊆ R ∪ C. The cost C s of a node s is the sum of the cost of the property of each relation of

T (c s =
Ts C(r j )). At each loop, the unexplored node with the lowest cost is selected to be explored. The first step of the algorithm is to test if this node is a goal node thanks to the GOALTEST function. To do so, a SPARQL query is constructed to test the correct variables instantiation. When such a query is submitted to an ontology, it will return all the combinations of variable substitutions. All these substitutions are stored for each node in a map M linking each variable to the entities that could be substituted to it (e.g. M(x t ) = {knif e1, knif e2, knif e3}). A node is thus a goal node if it respects the three constraints and if a t is the only substitution in M for the target variable x t (e.g. M(x t ) = {knif e2}). Then, if the node is not a goal node, meaning that the corresponding RE is flawed or ambiguous, we attempt to improve it by inserting a new relation to T thanks to the CREATEADDITION function. The relations costs assignation are not defined here. One can refer to [START_REF] Koolen | Learning preferences for referring expression generation: Effects of domain, language and algorithm[END_REF] or [START_REF] Belke | Tracking the time course of multidimensional stimulus discrimination: Analyses of viewing patterns and processing times during "same"-"different "decisions[END_REF] for property cost determination.

CREATEADDITION is the function that aims to create additions in order to generate new nodes. It has two dedicated sub-functions. The first one is the TYPINGADDITIONS function. For each subject and object a i of every relation r of T being anonymous entities, if it does not exist any relation of the form (a i , isA, t), we add such relation. The second sub-function is the DIFFERENCEADDITIONS function. This function is performed only if the first one could not create an addition for the current node (i.e. all relations already have a type). For each variable stored in M, it searches the divergent relations between all the possible substitutions and the entity represented by the variable. While the first function ensures that the RE will be verbalizable, the second ensures a reduction of the ambiguity at each loop.

To be able to refer to agents' past tasks, the GOALTEST function which assesses if a node is a goal node has to be modified. Now, in addition to test if M(x t ) = a t , it checks if every past tasks used in T are assigned with all their parameters. This constraint to get all the parameters of a task is important for the linguistic realization. If this constraint were not satisfied, we could have results where even the agent having performed the task would not be referred. Now we can test if a goal node involves all the parameters of a task, we expand the CREATEADDITION function to explore these parameters (Alg. 1). While the DIFFERENCEADDITIONS aims at exploring relations that differ from other ambiguous entities, the With ACTINGADDITIONS, for each variable x i of M having several substitutions, we search in K S all relations involving the anonymous entity a i and a task. With regard to the representation of §III-C, we are searching all the relations r = (a i , isP arameterOf, a task ) ∈ R ∧ (a task , HtnT ask) ∈ C.

The second added function, COMPLETIONADDITIONS, aims to ensure that any task used in a candidate RE has all its parameter inserted in that RE. For every task a task used in T , if a relation of R having for subject a task and for property hasP arameter is not present in T , this relation is added to the possible additions of the current node. The ACTINGADDITIONS function, as well as the DIFFERENCEADDITIONS, is performed only if the TYPINGADDITIONS and the COMPLETIONADDITIONS have not generated any possible addition. Through this condition, we ensure that every inserted task has all its arguments before adding other kinds of relation and that every inserted parameter has been typed before inserting a new one.

The cost of an addition representing a task (i.e. an addition involving the property hasP arameter or isP arameterOf ) is the cost of the task itself divided by the number of parameters. We chose to process in this way to avoid zerocost addition and because every inserted parameter will already have a cost due to at least their typing. Thanks to the episodic KB, the cost of these additions can also be weighted depending on the amount of time passed since the task has been performed. This meets the decay theory used in [START_REF] Williams | Toward forgetting-sensitive referring expression generationfor integrated robot architectures[END_REF] and makes a preference over the more recent which could be easier to remember for the RE receiver.

V. RESULTS

We present hereafter the solutions found by our algorithm on an illustrative example. Then, we discuss execution time measures to analyze the impact of such an extension.

A. Test cases

Let us take activities related to the HTN of Fig. 2. Based on them, a planner elaborates a shared hierarchical plan illustrated in Fig. 4. The primitive tasks are listed and organized according to a timeline. The abstract tasks hierarchy is shown in the tree on the right. The planner has assigned the tasks to two agents; the robot (Pr2) and a human (Alex). We introduce a third agent, the human Bob. We define five cases depending on the moments when Bob is in the kitchen where Pr2 and Alex are collaborating for the meal preparation. The colored lines, next to the timeline, represent the moments where Bob is in the kitchen for each case. For example, in case 1, Bob is only aware of the last Cut(Alex,tomato2,knife2) task and the CookSauce(Pr2, tomato2) task. The primitive tasks of which Bob is aware are thus added to his knowledge bases, both semantic and episodic. We consider that Bob is the spectator and is aware of the abstract tasks for which he is aware of all the tasks of their decomposition. Again in case 1, Bob is thus aware of the executed abstract tasks under PrepareSauce(). 4. The hierarchical execution trace to prepare the meal and a trace for another subsequent high-level task, organized according to a timeline. In the current instant, PR2 is asking Bob, a second human, for knife1. The five cases are represented by five colors at the left of the tasks and correspond to tasks seen by Bob, the human spectator. In case 1, Bob did not see any tasks involved in the preparation of the salad but was present during the sauce preparation. In the second case, he observed all the activities.

In all five cases, the goal is for PR2 to refer to the knife knife1 to Bob. We assume that the communication about it occurs out of the kitchen. This means that no spatial knowledge is available to generate the RE. In such a scenario, the robot would estimate that the Bob's semantic knowledge base does not contain any relations between knife1 and other objects. Thus, the approach presented in [START_REF] Buisan | Efficient, situated and ontology based referring expression generation for human-robot collaboration[END_REF] would not find any solution for any of the five presented cases. However, thanks to the contribution presented in this paper, a referring expression is found for four out of five cases.

Case 1) The algorithm first types knife1 as being a Knife. After this step, no additions about past tasks can be found by ACTINGADDITIONS since no task implying knife1 is known by Bob. Other relations, such as its color, can be added by the DIFFERENCEADDITIONS but do not solve the ambiguity. No solution is thus found.

Case 2) knife1 is first typed. This time all the tasks (primitive and abstract) are known by Bob. The three primitive tasks and two abstract tasks involve knife1 in their parameters. Five additions are thus generated in the form of (knif e1, isP arameterOf, a t ), where a t are the instances of the tasks. DIFFERENCEADDITIONS still not find additions here. In the five new nodes, the tasks are not labeled and are thus typed. Because every candidate RE under exploration here involves a task, the COMPLETIONADDITIONS function adds one new parameter for each through the hasP arameter property. These parameters are then typed if needed at the next step. Because the abstract task PrepareVegetableForSalad has only two parameters, the node exploring it is valid at this step, since the parameter of the knife has been added by the relation (knif e1, isP arameterOf, a t ) and the parameter of the agent has been added by the relation (a t , hasP arameter, Alex). Besides, for these two parameters, Knife1 is typed and Alex is not an anonymous entity. The candidate RE is: (?0 isA Knife), (?0 isParameterOf ?1), (?1 isA PrepareVegetableForSalad), (?1 hasParameter Alex). Matching it in the knowledge base gives only one substitution for the variable ?0 being the target entity knife1. It is thus a solution. The verbalization 1 would be "the knife with which 1 The proposed verbalization is automatically generated as proof of usability but the algorithm is not presented in this paper Alex prepared the salad vegetables".

Case 3) Bob is only aware of two primitive tasks involving knife1 being cutting tasks. The beginning of the process is the same as the previous case. The third parameter of each task is also explored and typed in a second time. At this step both active nodes are valid. Because both have the same number of parameters, their cost difference depends on the time. The node with the lowest cost is the one with the more recent task (i.e. the second cutting task). The candidate RE is : (?0 isA Knife), (?0 isParameterOf ?1), (?1 isA Cut), (?1 hasParameter Alex), (?1 hasParameter ?2), (?2 isA tomato). Only one substitution for the variable ?0 being the target entity. The node is thus a goal node. The verbalization would be "the knife with which Alex cut the tomato".

Case 4) Previously, even if Alex had cut another tomato with another knife, it did not lead to any ambiguity since it was not known by Bob. This time, Bob knows that Alex cut two different tomatoes with two knives so the previous solution is not a goal node anymore. The algorithm thus refers to the knife through the cut task on the cucumber (i.e. the first cutting task). The verbalization would be "the knife with which Alex cut the cucumber".

Case 5) The only task known by Bob and involving the knife1 is to cut a tomato. However, it leads to ambiguity with the other cut task with knife2. Despite this, our algorithm is able to find a solution in the way described previously, by choosing anyway the cut task and trying to specify the diverging argument that is the tomato. The tomato 1 being involved in the MixSalad task. The algorithm is able to use this second task to specify the argument of the first one and finally gives the RE: (?0 isA Knife), (?0 isParameterOf ?1), (?1 isA Cut), (?1 hasParameter Alex), (?1 hasParameter ?2), (?2 isA tomato), (?2 isParameterOf ?3), (?3 isA MixSalad), (?3 hasParameter Pr2), (?3 hasParameter ?4), (?4 isA Cucumber). The verbalization would be "the knife with which Alex cut the tomato that I mixed with the cucumber".

B. Performance analysis

To assess the impact of the proposed extension, we have chosen a realistically-sized knowledge base (101 entities, 36 classes, 40 properties, and 497 relations) already used in [START_REF] Buisan | Efficient, situated and ontology based referring expression generation for human-robot collaboration[END_REF]. It describes an apartment with three rooms including several pieces of furniture (tables, shelves) and objects (cups, boxes) linked through spatial relations (atLeftOf, onTopOf) and attributes (color, weight). However, no tasks are described in it. The original algorithm and its extended version have been run over all the 77 entities inheriting from the "Object" class, representing physical entities. The knowledge base is managed using the Ontologenius system [START_REF] Sarthou | Ontologenius : A long-term semantic memory for robotic agents[END_REF]. The extension has a negligible impact when no task is described in the ontology, with an average execution time of 1.04ms for the original algorithm versus 1.16ms for the extension. Even the most complex case, only changes from 6.25ms to 6.86ms.

In order to estimate the impact when tasks are present in the ontology, we chose to put ourselves in the worst case where tasks are described but not useful to find a solution. We add two actions with three parameters each for each object described in the apartment. This leads to an addition of 144 entities to the ontology (the 144 tasks) and 432 relations (three parameters per task). Such an amount of tasks could be the result of several hours of interaction. The added tasks have been designed such as not to help in the RE generation and just overload the algorithm. In this way, we have to find the same solutions as in the previous test. Here the impact of the extension is much more noticeable going, for the most complex entity, from 6.25ms to 245.25ms. We note that 75% of the entities are solved in 20.51ms versus 1.29ms, 50% in 5.08ms versus 0.51ms, and 25% in 0.58ms versus 0.18ms. We thus observe that, the longer the RE is to compute with the original algorithm the more noticeable the impact of the extension is. However, in an HRI context, even the worst case is still acceptable and will not spoil the interaction, as the response time is still under the user's flow of thought interruption time, and almost under the instantaneous feeling time [START_REF] Nielsen | Usability engineering[END_REF]. Moreover, we are here in the worst case where the added tasks are not useful to generate the RE. In more realistic cases, the use of past tasks in the REG may allow to reduce the solution length and hence the execution time.

VI. CONCLUSION AND FUTURE WORK

The main contribution of this paper is an extension to the original ontology-based REG which allows the robot to exploit the shared knowledge about past activities stored in a properly designed execution trace in order to produce RE involving an entity. The corollary contribution is a formalization of the representation, in an ontology, of any HTN and the execution trace of a plan built upon it.

We have already integrated the original algorithm within a cost-based human-aware task planner to allow it to assess the cost and feasibility of communication actions containing references to entities. We intend now to devise a planner which will use this new capability to generate pertinent communication actions involving "future past tasks" contexts.

Fig. 1 .

 1 Fig. 1.Referring to knife 2 in the current situation (b) is impossible if the robot is performing an action that does not allow it to see what is in front of the human. Considering a previous step of the human's task, the robot can refer to the knife through the action to cut a tomato (a).

Fig. 2 .

 2 Fig. 2.The domain of the high-level task PrepareMeal. It is used by the planner (HATP) to elaborate an HR shared plan through a contextbased decomposition and parameter instantiation process (which vegetable V, which knife K, ...) including the selection of which agent A (Alex the human, or PR2 the robot) subtasks and/or actions will be allocated.

:

  PrepareVegetable rdf:type owl:Class ; :PrepareVegetable rdfs:subClassOf :HtnAbstractTask ; :PrepareVegetable :hasDecomposition :PVDecomp_A , :PrepareVegetable :hasDecomposition :PVDecomp_B . :PrepareVegetableDecomp rdf:type owl:Class ; :PrepareVegetableDecomp rdfs:subClassOf :HtnDecomposition. :PVDecomp_A rdf:type owl:Class ; :PVDecomp_A rdfs:subClassOf :PrepareVegetableDecomp ; :PVDecomp_A :hasSubtask :Cut_PvDecomp_A , :PVDecomp_A :hasSubtask :Peel_PvDecomp_A . :Cut_PvDecomp_A rdf:type owl:Class ; :Cut_PvDecomp_A rdfs:subClassOf :Cut ; Description 1. Description of the abstract task PrepareVegetable and one of its decomposition. Let us consider the abstract task PrepareVegetable of the domain illustrated in Fig. 2. The generated OWL representation (Desc. 1) is composed of the PrepareVegetable class inheriting from the HtnAbstractTask concept. Through the properties hasDecomposition, we state that the task has two decompositions being PVDecomp A and PVDecomp B.

:

  Cut_hasParameter.A rdf:type owl:ObjectProperty ; :Cut_hasParameter.A rdfs:subPropertyOf :hasParameter.0 ; :Cut_hasParameter.A rdfs:domain :Cut ; :Cut_hasParameter.A rdfs:range :Agent . :Cut_hasParameter.V rdf:type owl:ObjectProperty ; :Cut_hasParameter.V rdfs:subPropertyOf :hasParameter.1 ; :Cut_hasParameter.V rdfs:domain :Cut ; :Cut_hasParameter.V rdfs:range :Vegetable . Description 2. Description of hasParameter property specifications for the first and second parameters (resp. the agent performing the task and the cut vegetable) of the Cut primitive task

:

  Decomp_PV_3 rdf:type owl:NamedIndividual , :Decomp_PV_3 rdf:type :PVDecomp_A ; :Decomp_PV_3 :DecompositionUsedBy :PV_3 ; :Decomp_PV_3 :hasSubtask :Peel_5 , :Decomp_PV_3 :hasSubtask :Cut_7 . :Cut_7 rdf:type owl:NamedIndividual , :Cut_7 rdf:type :Cut_PVDecomp_A ; :Cut_7 :Cut_hasParameter.A :Alex ; :Cut_7 :Cut_hasParameter.V :cucumber1 ; :Cut_7 :Cut_hasParameter.K :knife1 ; Description 3.

Fig.

  Fig.4. The hierarchical execution trace to prepare the meal and a trace for another subsequent high-level task, organized according to a timeline. In the current instant, PR2 is asking Bob, a second human, for knife1. The five cases are represented by five colors at the left of the tasks and correspond to tasks seen by Bob, the human spectator. In case 1, Bob did not see any tasks involved in the preparation of the salad but was present during the sauce preparation. In the second case, he observed all the activities.

  ACTINGADDITIONS complements it by exploring the tasks in which the entities involved in the candidate RE are part of.

	Algorithm 1 The modified CREATEADDITION function
	function CREATEADDITION(node)
	additions ← TYPINGADDITIONS(node)

if additions = ∅ then return additions additions ← COMPLETIONADDITIONS(node) if additions = ∅ then return additions additions ← DIFFERENCEADDITIONS(node) additions ← additions ∪ ACTINGADDITIONS(node) return additions

Acknowledgement: This work has been funded by the Agence Nationale de la Recherche JointAction4HRI project ANR-16-CE33-0017 and the Artificial and Natural Intelligence Toulouse Institute (ANITI).


Baptiste Darnala
email: baptiste.darnala@elzeard.co

Florence Amardeilh
email: florence.amardeilh@elzeard.co

Catherine Roussey
email: catherine.roussey@inrae.fr

Clément Jonquet
email: jonquet@lirmm.fr

Crop Planning and Production Process Ontology (C3PO), a new model to assist diversified crop production

Keywords: ontology, agriculture, semantic resource, knowledge representation, crop itinerary IFOW'21: Integrated Food Ontology Workshop, September 15-18, Bolzano Italy

Crop planning and production process ontology (C3P0) is a representation of agricultural knowledge for diversified crop production. It is composed of eight modules to take into account all the elements involved in crop production as well as the complexity for representing farming practices and constraints. Especially, the crop management module is sliced in three different layers to capture the generic information about crop itineraries, the planned production process and the tasks eventually really achieved been by various agents in the plots. To implement this ontology, we designed a new pipeline, presented here, integrating various existing tools to ease the ontology documentation, population and maintenance tasks.

Introduction

Agricultural production practices are changing and crop production are facing multiple vulnerabilities: economic, regulatory, environmental and social (isolation, skills, overload, etc.). Moreover, the pressure created by distribution chains and the consumers' demand for better and healthier vegetables, produced in local area, force farmers to improve crop management. Several scientific studies [START_REF] Isbell | Benefits of increasing plant diversity in sustainable agroecosystems[END_REF][START_REF] Paut | Reducing risk through crop diversification: An application of portfolio theory to diversified horticultural systems[END_REF] have shown diversity --both in time and space-of cultivated plants:

• improves risks management with respect to weather and economic changing context; • delivers a natural defence system against diseases and pest attacks; • increases stability and resilience of agro-ecosystems.

However, with new agroecological techniques, farmers' workload, management complexity and mental load also increase [START_REF] Morel | Strategies to manage crop planning complexity in very diversified direct selling farming systems: the example of organic market gardeners[END_REF][START_REF] Dumont | Why working conditions are a key issue of sustainability in agriculture? a comparison between agroecological, organic and conventional vegetable systems[END_REF]. Between 2019 and 2020, the SME Elzeard conducted over 150 interviews with farmers, agricultural advisors, teachers and researchers. The company identified several technological barriers, among them the need for knowledge sharing and for new operational tools to assist farmers in their daily crop management: optimizing their crop rotations, their yields and finding alternatives to chemicals.

To address the previous identified need, Elzeard develops a web and mobile application to assist farmers in their planning and production activities. In order to encode the knowledge in the system and to provide right decision-support and adapted recommendations, Elzeard have designed an ontological model for representing plot management and crop itineraries, called Crop Planning and Production Process Ontology (C3PO). This work is done in collaboration with the ANR D2KAB project consortium 1 and the MesclunDurab consortium 2 . The objective of this ontology is to structure a knowledge graph about agroecological crop practices, that will become a common good for the benefits of the various actors of the vegetable production sector. The knowledge graph will be accessible via a Web portal, called La Serre des Savoirs being currently developed. La Serre des Savoirs will leverage the information contained in the knowledge base into a more usable way for the different actors of the food industry. The C3PO ontology will also be published and shared on the AgroPortal ontology repository [START_REF] Jonquet | Agroportal: an ontology repository for agronomy[END_REF] and adhere to the FAIR principles (Findable, Accessible, Interoperable, Reusable) [START_REF] Mark | The fair guiding principles for scientific data management and stewardship[END_REF].

In this paper, we present preliminary results about the design requirements and implementation workflow of the C3PO ontology. Section 2 covers related research regarding crop production models; Section 3 explains the specificity of the diversified vegetable crop production representation; Section 4 outlines the first version of the C3PO ontology; Section 5 describes the ontology creation pipeline used to develop, document and populate the ontology with reference data.

Related work

Semantic Web technologies were chosen to represent the interlinked knowledge needed to manage crop planning and production process in its digital application because of their two key advantages. Not only do these technologies allow knowledge to be represented through formal and logical semantics enabling native reasoning (the ontology), but they also allow this knowledge to be interconnected with open linked data thanks to shared or aligned ontologies, thus promoting data interoperability (graph database).

For designing the C3PO ontology, we firstly examined existing ontologies [START_REF] Drury | A survey of semantic web technology for agriculture[END_REF] that could be reused to represent plant characteristics and crop itineraries, that is to say general information on a sequence of farming practices to ensure optimal growth of all crops in a plot. Some agronomic resources about plants or crops are already available in a semantic format like the agroecological knowledge base GECO [START_REF] Soulignac | Geco, the french webbased application for knowledge management in agroecology[END_REF] or in various thesaurus and agronomic reference ontologies like Plant Ontology [START_REF] Avraham | The plant ontology database: a community resource for plant structure and development stages controlled vocabulary and annotations[END_REF], Crop Ontology [START_REF] Arnaud | Towards a reference plant trait ontology for modeling knowledge of plant traits and phenotypes[END_REF], AGROVOC [START_REF] Caracciolo | The agrovoc linked dataset[END_REF], TAXREF-LD [START_REF] Michel | A model to represent nomenclatural and taxonomic information as linked data. application to the french taxonomic register, taxref[END_REF] or FrenchCropUsage [START_REF] Roussey | Frenchcropusage: Thésaurus sur les cultures françaises. le thésaurus décrivant les cultures françaises par leur utilisation au format skos[END_REF]. Nevertheless, these semantic resources did not describe farming practices and their interactions from an agricultural production perspective. Searching AgroPortal's content for "crop management plan" or "crop itinerary", returned several results. The first one is the Durum Wheat ontology but it only contains "itinerary" as a < s k o s : C o n c e p t > , i.e. as a terminological definition. We also found a "TechnicalItinerary" as a < o w l : C l a s s > into MS2O but this ontology is now deprecated. Another ontology identified was AgroRDF [START_REF] Martini | agrordf as a semantic overlay to agroxml: a general model for enhancing interoperability in agrifood data standards[END_REF]. It represents and describes the farm practices as a set of technical processes. Unfortunately some of their classes are only defined in German and there is no sequence of processes or plan.

Lately we also discovered Agronomy Ontology (AGRO) [START_REF] Devare | Data-driven agricultural research for development: A need for data harmonization via semantics[END_REF]. The farming practices are defined as a set of technical processes, named planned processes, in the context of agronomic experiments. This ontology is based on Basic Formal Ontology (BFO) [START_REF] Arp | Building Ontologies With Basic Formal Ontology[END_REF]. The main scope of application of BFO seems to focus on the concrete representations of the real world and not really on the abstract representations of the human mind that may help in decision making. That is why AGRO is oriented towards the recording of precise observations concerning specific agronomic experiments rather than towards the planning of agricultural production from reference data. We are already planning future work to interconnect the definitions of common F a r m i n g p r a c t i c e s with those of the Agronomic Ontology.

We also looked for for process representation from other domains, such as the W3C Recommendation Prov Ontology [START_REF] Lebo | PROV-O: The PROV Ontology, W3C Recommendation[END_REF]. Prov-O is a process-flow model in the sense that it traces the provenance and evolution of A c t i v i t y concepts, interacting with Agents and other Entities. It is a very interesting model because it brings a representation of agents, activities but also entity collection, ordered or not, that can be very useful to represent a vegetable production. Indeed, with Prov-O, we could represent the farm, its actors represented by the producers for example, and the different activities realised on plants or in the plots. But after testing its implementation on our use cases, we chose not to use Prov-O to represent crop planning and production process as it does not allow a theoritical representation of activities with relative dates placed into the calendar. We rather decided to use Prov-O to keep trace of the updates on the various entities in the knowledge graph data.

We studied another process ontology dedicated to transformation processes in food science, called Process and Observation Ontology (PO2) [START_REF] Liliana Ibanescu | Juliette Dibie-Barthelemy, Po² -a process and observation ontology in food science. application to dairy gels[END_REF]. This ontology is also derived from BFO principles and has the same drawbacks that AGRO. It was a good start for implementing operational processes composed of steps, products, observations, measures, and agents. But it was not adapted to distinguish the reference data about the crop itineraries from the observed data in a specific plot.

Recently, we discovered Valueflows3 , an ontology that describes flows of economic value according to a three layers representation (Knowledge, Plan, Observation). As described in Section 4, we used the same three-layers division to design the ontology part about crop management process. To the best of our knowledge, this is the first conceptualisation of a sequence of cultivation processes associated with a production plan linked to some generic knowledge about those plans.

Representing diversified vegetable production

The workflow of vegetable production and farm management requires a high level of knowledge with regards to production techniques, especially to adapt to the on-going agroecological transition. Moreover, with increasing regulatory standards and consumer pressure, it becomes very complicated for farmers to produce diversified, tasty and healthy vegetables without too many economic losses. It is therefore necessary to relieve the mental burden of farmers by identifying all the knowledge related to agricultural techniques that allow them to organise, in time and space, the schedules of each of the crops as well as their association and rotation over a given season.

To face this challenge, the digital services offered by Elzeard to vegetable farmers rely on the exploitation of the technical crop itineraries. A crop itinerary can be defined as a generic process consisting of a succession of technical tasks and recommendations for agricultural production. For a given crop, the crop itinerary has to be specialised according to the cultural context and the farmer's cultivation methods. Elzeard's decision support system assists farmers in the adaptation from a generic crop itinerary to a custom-made one, therefore optimizing crop production.

The scope of the Crop Planning and Production Process Ontology (C3PO) presented in this paper therefore targets information about the various possible crop itineraries and their dependence with the whole agro-ecosystem. It has to take into account the general information about the crops and cultivars but also the constraints about the farming system (organic or conventional), the location of the farm along with the soil and climate conditions, the client orders, the crop rotations and associations, the farming equipments, the potential diseases or aggressors and crop protection solutions, etc.

Ontology modelling

As the scope of this ontology is quite large and complex, we have chosen to split it into eight modules where each part defines a particular need of crop production. The modules interact with each other to represent the entire planning and production process on a farm, from seed selection to crop delivery. Moreover, in order to be interoperable according to FAIR principles, the modules have been linked to external ontologies for direct reuse or alignment wherever possible as described in Figure 1.

• The Plant module represents cultivated plants (e.g., "Carrot" vegetables), their botanical family (e.g., "Apiaceae"), their usage category (e.g., "Root vegetable") and the different cultivars (e.g., "Nantaise") with their characteristics like the earliness, size, colour, shape. This module reuses the French Crop Usage thesaurus [START_REF] Roussey | Frenchcropusage: Thésaurus sur les cultures françaises. le thésaurus décrivant les cultures françaises par leur utilisation au format skos[END_REF] for vernacular names, crop usages and aligns with TAXREF-LD [START_REF] Michel | A model to represent nomenclatural and taxonomic information as linked data. application to the french taxonomic register, taxref[END_REF] for scientific names and taxonomic organization of families and species and plant development stage with BBCH (Biologische Bundesanstalt, Bundessortenamt und CHemische Industrie) 4 . • The Aggressor module defines potential diseases and pests of each cultivated plants with their description and protection solutions. For this we will rely on the GECO (GEstion des COnnaissance) 5 ontology model and knowledge base. We are working with the GECO's team in D2KAB to document the model and align it with other resources such as Agrovoc, Wikidata or TaxRefld to complete with additional data about those living organisms. • The Treatment module contains the description of every inputs that enhances plant growth or plant defence. For this we will rely on an OWL version of French ANSES's E-Phy inputs catalogue developed in D2KAB. • The Plot module represents the spatial organisation of the farm, the division into plots and even beds for diversified vegetable production, the different infrastructures present on the farm like the irrigation equipment, etc. For this we have not yet identified an external semantic resource that goes into sufficient details regarding the plot description dedicated to diversified crop productions. • The Admin module imports and links with existing ontology models such as FOAF, Time Ontology, the Event Ontology or Prov-O, to represents agents, organisation or administrative entities. • The Supply Manager module organises the stocks and the product delivery. For this module, we are discussing with the DataFoodConsortium6 to reuse part of their supply chain ontology model, which they are currently also aligning on ValueFlows model. • The Crop Management module represents all the crop production process. It is the core of the C3PO ontology, importing all other modules needed to organise the crop production.

• The Vocabulary module, formalized in SKOS, organises reference values used for the definitions of class instances, like units of measures (aligned with QUDT definitions), the climate choices, the farming practices, the earliness levels and so on. Each time it is possible, reference values are aligned to AGROVOC concepts.

The Crop Management module is the most complex. To better take into account the crop planning and production process, we need to represent crop itineraries from different perspective: the generic theoretical information as advised in agricultural literature -how tomatoes should be grown under various constraints; the planned process -how a farmer plan to produce tomatoes according to its own constraints; the observation data -how a farmer eventually produced tomatoes taking into account possible hazards.

Those three dimensions better capture changes between plans and results in order to optimise production from one year to another. Additionally, we want to encourage the sharing of observation data between farmers to update theoretical knowledge if needed to propose adaptations to climate change for example.

To describe the process according to those separate layers we get inspiration from the Value-Flows model, itself inspired by the REA ontology [START_REF] Geerts | The ontological foundation of rea enterprise information systems[END_REF] about economic phenomena occurring in Enterprise Information System. REA introduced a distinction between Knowledge Infrastructure and Operational Infrastructure whereas ValueFlows model goes further distinguishing the Knowledge level (core concepts, guidelines, recipes, procedures), the Plan level (offers, requests, schedules and plans) and the Observation level (what really happened). This threelayer representation completely fits our requirements for modelling the crops' operational planning and production process. Figure 2 describes the production flow with an example. The Knowledge Level describes the generic concepts about plants, technical operations that are part of a itinerary procedure, implementing specific farming practices organised as a hierarchy, the various contextual conditions that may affect the expected yield, the relative periods of the different steps of the process (germination, plantation, maintenance, harvest, conservation), etc. Then the Plan Level represents the production process that is deduced from the itinerary procedure, the cultivars used as a seed material, the planned operational tasks, from soil preparation to harvest, according to market opportunities, delivery dates and volumes, etc. Moreover, in a diversified crop production, such as for the vegetables, it is important to take into account the fact that a production process can be sliced into various series of the same crop in order to have longer lasting harvests. A farmer repeats more or less the same tasks of a production process, at different dates to benefit from a continued flow of production. Lastly, the Observation Level declines the planned operational tasks into activities that are assigned to a specific location (plots and/or beds), persons or teams and at a certain date. It gives an historical view of the achieved task with complementary data about the equipment used, the inputs dose or volumes, the time spent to do the activity, etc. It also represents observation data (weather conditions, hazards, presence of an agressor, and so on).

Pipeline of ontology production

To start implementing our ontology, we used the Linked Open Terms (LOT) [START_REF] Poveda-Villalón | A reuse-based lightweight method for developing linked data ontologies and vocabularies[END_REF] methodology as it proposes an agile and iterative approach to ontology development. First, we worked with We gathered additional requirement specifications from various domain-oriented text books, guidelines procedures, crop datasets or dedicated knowledge bases such as GECO. We also chose to reuse existing ontologies and thesauri in order to avoid reinventing the wheel and facilitate the interoperability of our model, either upper-level metadata vocabularies such as FOAF, EVENT Ontology or OWL-TIME or domain-specific ontologies such as FCU or TAXREF-LD.

At first, to model the onotlogy we used a graphical model by using the online graphic tool "Draw.io"7 to ease the discussions with the domain experts. Then, we had to manually transcript the concepts' definitions of this graphical model in an ontology editor such as Protégé [START_REF] Musen | The protégé project: a look back and a look forward[END_REF]. Not only this method is prone to human error when transcribing the ontology model but it may lead to serious problems when keeping the graphical model up to date, still useful to provide an overview of the ontology model. It appeared that some modifications made into Protégé were not always updated on the graphical model. To counter this problem, an ontology creation pipeline was built by combining different existing tools in the same workflow. The first tool of that pipeline is still Draw.io but we added Chowlk [START_REF] Chavez-Feria | Converting uml-based ontology conceptualizations to owl with chowlk[END_REF], a graphical extension for an ontology representation. Thanks to a standardized graphical notation to represent ontological classes, properties and restrictions, Chowlk automatically transcribes Draw.io's XML output into an OWL representation with the Turtle syntax. Then, the generated TTL file can be imported into Protégé for verification and validation by the ontologist. Then the ontology model can be imported in a RDF triple store. GraphDB8 has been chosen for its performance and reasoning capabilities as the RDF storage of the future Knowledge Base. Moreover, to simplify the documentation process, GraphDB includes OntoRefine, an integrated upgraded version of the open-source tool OpenRefine9 . It enables data transformation for converting tabular data (e.g. CSV) into RDF data. There are two uses for this tool. The first one is to import the concepts and properties definitions (all labels, comments and other annotation properties) that are maintained outside the Draw.io to make the graphical model easier to read. The second one is to populate the data graph with knowledge provided by domain experts in tabular files as it is often more convenient for them. Once the ontological model is complete, the last tool used is Widoco [START_REF] Garijo | Widoco: A wizard for documenting ontologies, The Semantic Web -ISWC 2017 10588[END_REF] to automatically generate all the ontology documentation.

The C3PO ontology documentation is currently being finalised and will be published at its namespace URL 10 . Meanwhile, the C3PO's owl files are available at Serre des Savoirs project's GitLab repository 11 to collect future contributions and revision requests.

Conclusion

To comply with operational needs for Elzeard's digital application, we modelled a new ontology for representing Crop Planning and Production Process dedicated to diversified crop production. Even if we tried to reuse existing thesaurus and agronomy ontologies, there was no existing ontology that captured the specific need of representing crop itineraries from the perspective of the farmers' production processes. The C3PO ontology is used to construct a knowledge base for the storage of the Elzeard web application data and into la Serre des Savoirs, a web portal that will give access to extensive knowledge about plants and their crop itineraries on a practical way for the agriculture actors. Various domain expert partners were included in our approach to question and identify the main concepts and properties: scientists (D2KAB, MESCLUNDURAB) and agricultural professionals (crop farmers, networks of agricultural advisors, teachers.). A pipeline was initiated and tested to ease the different steps of ontology creation, documentation and population, integrating various independent tools. In future work, we will align our concepts and properties to BFO models, such as Agronomy Ontology or PO2. We will also publish the ontology model according to the FAIR principles on AgroPortal and our GitLab for tracking issues and new contributions. For the first time, an ontology with information oriented on French farming practices is developed, both in French and in English. However, the ontology is intended to become a global open knowledge base about all farming practices and crop itineraries related to diversified crop production such as market gardening, arboriculture or agroforestry.

Figure 1 :

 1 Figure 1: Overview of the Crop Planning and Production Process Ontology.

Figure 2 :

 2 Figure 2: Example of a production process with concepts represented in the three layers of the C3PO module

Figure 3 :

 3 Figure 3: C3PO's pipeline for ontology production.

http://www.d2kab.org/

https://maraichage.wixsite.com/mesclun

https://lab.allmende.io/valueflows

http://ontology.irstea.fr/bbch/core

https://geco.ecophytopic.fr/

https://www.datafoodconsortium.org/en/

https://www.draw.io

https://www.ontotext.com/graphdb

https://openrefine.org/

http://www.elzeard.co/ontologies/c3po

https://gitlab.com/serre-des-savoirs/c3po

Acknowledgments

This work was partially achieved with support of the Data to Knowledge in Agronomy and Biodiversity (D2KAB -www.d2kab.org) project that received funding from the French National Research Agency (ANR-18-CE23-0017). We also acknowledge support from the National Office for Biodiversity with MesclunDurab grant and the Nouvelle-Aquitaine Region with "Social Innovation AMI" and "Digital Prototypes" grants. We also thank Dr. Kevin Morel for his feedback and all contributors from MesclunDurab and D2KAB projects.