Cezary Zieliński
email: c.zielinski@ia.pw.edu.pl

Maciej Stefańczyk

Tomasz Kornuta

Maksym Figat

Wojciech Dudek

Wojciech Szynkiewicz

Wlodzimierz Kasprzak

Jan Figat

Marcin Szlenk

Tomasz Winiarski

Wlodzimierz Kasprzak

Włodzimierz Kasprzak

Konrad Banachowicz

Teresa Zielińska

Emmanouil G Tsardoulias

Andreas L Symeonidis

Fotis E Psomopoulos

Athanassios M Kintsakis

Pericles A Mitkas

Aristeidis Thallas

Sofia E Reppou

George T Karagiannis

Konstantinos Panayiotou

Vincent Prunet

Manuel Serrano

Stratos Arampatzis

Alexandros Giokas

Lazaros Penteridis

Ilias Trochidis

David Daney

Miren Iturburu

Variable structure robot control systems: The RAPP approach

Keywords: robot controllers, variable structure controllers, cloud robotics, RAPP

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

World population is ageing [START_REF] Lutz | The coming acceleration of global population ageing[END_REF] and in Europe this process is especially acute.

The process of aging in an individual inevitably leads to a decline in both the mental and physical abilities and that usually results in at least partial social exclusion. To prevent this, a plethora of social workers could be employed, nevertheless the ratio of working-age people decreases in relation to the number of the elderly. Thus an alternative remedy will have to be employed. It is believed that the reintroduction of the infirm into society can be facilitated by assistive robots [START_REF] Reppou | Social inclusion with robots: A RAPP case study using NAO for technology illiterate elderly at ormylia foundation[END_REF][START_REF] Reppou | RAPP: A robotic-oriented ecosystem for delivering smart user empowering applications for older people[END_REF][START_REF] Manzi | Design of a cloud robotic system to support senior citizens: the kubo experience[END_REF][START_REF] Bonaccorsi | Design of cloud robotic services for senior citizens to improve independent living in multiple environments[END_REF][START_REF] Deegan | Mobile manipulators for assisted living in residential settings[END_REF][START_REF] Kędzierski | Design for a robotic companion[END_REF][START_REF] Görer | An autonomous robotic exercise tutor for elderly people[END_REF][START_REF] Tsardoulias | Towards an integrated robotics architecture for social inclusion-the rapp paradigm[END_REF]. Robot companions can physically assist people by carrying out tasks that are either beyond their capabilities or simply difficult to accomplish by them, e.g. can perform detection of hazards at home, monitor the owner's health, ensure their safety and entertain them.

Those enhanced capabilities should reverse the trend of withdrawal from (or perhaps rejection by) the society.

A robot companion must be universal to cater to all the needs of the owner. This requires that both the mechanical construction and the control system of such a robot should be capable of executing very diverse tasks. This paper concentrates on the control system aspect of designing such devices. It should be noted that the requirements of the owner of the robot can be potentially unlimited, while its resources are definitely constrained. The size of the robot should not be too large, in order not to disconcert people, not to mention an increase of probability of potential injury due to inevitable collisions. As modern robots are computer controlled one should take into account that the reduction of the overall dimensions influences the size of the computer and thus its memory resources, which store the control software. Hence potentially unlimited requirements have to be matched to limited software resources. The solution of this problem is based on the idea of supplementing the limited on-board resources of the robot by the theoretically unlimited resources provided by the cloud. However this implies that portions of the control software will be downloaded during operation of the robot from the cloud. Moreover this software will need to assume control of the robot as only it has the capability ("knowledge") of solving the task formulated by the owner. The controller is always governed by the necessities of the task. Thus not only the software composition of the controller will vary but also the supervisory responsibilities will have to be transferred between its modules. This paper focuses on the solution of this problem. For that purpose a RAPP system is proposed, where RAPP is an acronym standing for: Robotic Applications for Delivering Smart User Empowering Applications [START_REF] Psomopoulos | Rapp system architecture[END_REF].

The paper is organized as follows. It starts with the presentation of robot control architectures (sec. 2), taking into account diverse criteria, so as to point out the specificity of a RAPP based robot controller. The RAPP system structure is based in the concept of an agent, which is briefly described in sec. [START_REF] Reppou | RAPP: A robotic-oriented ecosystem for delivering smart user empowering applications for older people[END_REF]. The description of a RAPP system architecture starts with its structure (sec. 4).

Then, the functions of particular RAPP agents are presented in sec. 5. The whole system was implemented on various robots (sec. 6) and experimentally verified with a hazard detection application (sec. 7).

Robot Control Architectures

The papers on robot control, although abundant, rarely specify the architecture in other terms than general text description, sometimes supplemented by block diagrams. There is no common way of representing architectures, not to mention universally accepted formal tools for that purpose. Although some work on formal specification of robot control software, e.g. [START_REF] Lyons | A formal model of computation for sensorybased robotics[END_REF][START_REF] Lyons | Adaptive behavior and intelligent systems without symbols and logic of Studies in cognitive systems[END_REF][START_REF] Zieliński | A Quasi-Formal Approach to Structuring Multi-Robot System Controllers[END_REF][START_REF] Zieliński | Diagnostic requirements in multi-robot systems[END_REF][START_REF] Armbrust | Soft robot control with a behaviour-based architecture[END_REF], and its formal verification verification, e.g. [START_REF] Kiekbusch | Formal verification of behaviour networks including sensor failures[END_REF][START_REF] Kiekbusch | Formal verification of behaviour 36 networks including hardware failures[END_REF], unfortunately it has not gained widespread acceptance. Thus, some form of general classification of robot control system structures (architectures) has to be produced to be able to compare diverse approaches to controller design. The overviews [START_REF] Kortenkamp | Robotic systems architectures and programming[END_REF][START_REF] Coste-Maniere | Architecture, the backbone of robotic systems[END_REF] distinguish:

• architectural structure, i.e. division of the considered robot control system into subsystems as well as the presentation of their interconnections, and

• architectural style, where computational and communication concepts are described.

However the paper [START_REF] Kortenkamp | Robotic systems architectures and programming[END_REF] points out that in many implemented systems it is difficult to clearly define their architectural structure and style. Since the time of that publication not much has changed. Some robot-related classifications have been mentioned in surveys, e.g. [START_REF] Matarić | The Handbook of Robotics[END_REF][START_REF] Bulter | Distributed and cellular robots[END_REF][START_REF] Parker | Multiple mobile robot systems[END_REF][START_REF] Yim | Modular self-reconfigurable robot systems [grand challenges of robotics[END_REF][START_REF] Matarić | Issues and approaches in the design of collective autonomous agents[END_REF][START_REF] Farinelli | Multirobot systems: a classification focused on coordination[END_REF][START_REF] Dudek | A taxonomy for multiagent robotics[END_REF][START_REF] Doriya | A brief survey and analysis of multirobot communication and coordination[END_REF][START_REF] Chibani | Ubiquitous robotics: Recent challenges and future trends[END_REF], however although they use similar criterions their definitions are not always consistent or evident (e.g. [START_REF] Matarić | The Handbook of Robotics[END_REF][START_REF] Bulter | Distributed and cellular robots[END_REF][START_REF] Parker | Multiple mobile robot systems[END_REF][START_REF] Yim | Modular self-reconfigurable robot systems [grand challenges of robotics[END_REF][START_REF] Matarić | Issues and approaches in the design of collective autonomous agents[END_REF][START_REF] Farinelli | Multirobot systems: a classification focused on coordination[END_REF][START_REF] Doriya | A brief survey and analysis of multirobot communication and coordination[END_REF]) or the classification criterions are marginally relevant to the topic discussed in this paper (e.g. [START_REF] Dudek | A taxonomy for multiagent robotics[END_REF]), so we need to disclose how we understand the criterions we use.

A crude definition of a robot can be provided from the perspective of its components. For our purposes we shall define a robot as a device having one or more effectors and zero or more receptors, controlled in such a way as to execute the allotted tasks, e.g. [START_REF] Zieliński | Reaction based robot control[END_REF][START_REF] Zieliński | A Quasi-Formal Approach to Structuring Multi-Robot System Controllers[END_REF][START_REF] Zieliński | Motion generators in MRROC++ based robot controller[END_REF]. Effectors are those devices that influence the environment in which the robot acts. For practical purposes, in robotics two categories of receptors are distinguished: exteroceptors and proprioceptors. The former gather the information from the environment and the latter provide data about the state of the effectors. Herein only exteroceptors will be of interest, thus for the purpose of brevity they will be called simply receptors. The control system acquires the information from the receptors and commands the effectors.

The task can be embedded in the control system or can be exchangeable.

Robotic systems are inherently complex. To overcome this complexity some kind of decomposition is necessary, thus control systems are usually divided into subsystems. In the case of a single-robot control system those subsystems in conjunction exert influence over the robot hardware (i.e. effectors and receptors) in such a way as to achieve the desired goal, i.e. execute the allotted task.

Examples of such single-robot multi-effector systems are abundant, e.g. [START_REF] Zieliński | Motion generation in the MRROC++ robot programming framework[END_REF][START_REF] Kędzierski | Design for a robotic companion[END_REF][START_REF] Dietrich | Reactive wholebody control: Dynamic mobile manipulation using a large number of actuated degrees of freedom[END_REF][START_REF] Likar | Virtual mechanism approach for dual-arm manipulation[END_REF][START_REF] Leidner | Knowledge-enabled parameterization of whole-body control strategies for compliant service robots[END_REF]. In the case of multi-robot systems the controllers of individual robots can act:

• separately (robots having separate control systems, acting unallied or doing work together, but without explicitly communicating with each other, e.g. one thousand robots filling-in a predefined 2D space [START_REF] Rubenstein | Programmable self-assembly in a thousand-robot swarm[END_REF]; two boxpushing robots perceiving the environment to accomplish a task [START_REF] Zieliński | Stigmergic cooperation of autonomous robots[END_REF]; a robot using a motion capture system as a remote sensor, guiding a formation of robots which observe through their own sensors the relative distance to the leader, which moves along a predefined path [START_REF] Tron | Vision-based formation control of aerial vehicles[END_REF]),

• together by explicitly exchanging information (individual robot controllers directly exchange information between themselves, e.g. flocking or foraging robots directly exchanging information [START_REF] Matarić | Issues and approaches in the design of collective autonomous agents[END_REF]; three-robot system based on ALLIANCE architecture communicating with each other to inform about the waste sites that each of them is taking care of [START_REF] Parker | ALLIANCE: An architecture for fault tolerant multirobot cooperation[END_REF]),

• together by being commanded by a supervisor (a distinguished supervisor influences the robot controllers through direct communication, e.g. supervisor of a multi-robot-based reconfigurable fixture supporting flexible workpieces during machining [START_REF] Zieliński | Control and programming of a multi-robot-based reconfigurable fixture[END_REF]; a system composed of a supervisor commanding three heterogeneous robots in conjunction carrying a rigid beam [START_REF] Simmons | First Results in the Coordination of Heterogeneous Robots for Large-Scale Assembly[END_REF]; master-slave approach tested both in simulation and field with four Stalker XE aerial vehicles flying in a formation [START_REF] Whitzer | In-flight formation control for a team of fixed-wing aerial vehicles[END_REF]).

It should be noted that the distinction between a single-and a multi-robot system is not that obvious. A two-hand device can either be treated as a single two-handed robot (e.g. two industrial manipulators holding an egg rigidly [START_REF] Zhu | Control of two industrial manipulators rigidly holding an egg[END_REF] or solving a Rubik's cube puzzle [START_REF] Zieliński | Rubik's cube as a benchmark validating MRROC++ as an implementation tool for service robot control systems[END_REF]) or two separate manipulators. In the former case a single controller exerts influence over the two manipulators, while in the latter case two separate controllers govern the actions of the manipulators separately. Hence how to differentiate between supervised two separate control systems of two manipulators and a single control system of a two-handed sys-tem? This dilemma is resolved by stating how the interactions between system components are treated. If the connections of the subsystems are treated as internal to the system, a single controller results, but if some of the connections are treated as external then several interacting systems are produced. In practice this resolution, based on the subjective treatment of interactions, usually coincides with the frequency of those interactions. If the required frequency is high, a single control system of the two hands will be required. If, on the other hand, the frequency is low, separate controllers of each of the arms commanded by an extra supervisor are an option (e.g. [START_REF] Zieliński | Control and programming of a multi-robot-based reconfigurable fixture[END_REF]). In the case of walking machines, which are single robots with multiple effectors, usually a single hierarchic controller is implemented, e.g. as in the case of quadrupeds BISSAM [START_REF] Berns | Mechanical construction and computer architecture of the four-legged walking machine bisam[END_REF] and LAVA [START_REF] Zielińska | Development of walking machine: mechanical design and control problems[END_REF][START_REF] Zielińska | Multifunctional quadruped[END_REF].

Coordination appears when a single control system governs the actions of one or more effectors or robots as well as when separate controllers communicate directly with each other to exchange coordinating information. Uncoordinated systems are composed of robots with separately acting controllers. Coordination can be continuous or sporadic. Continuous coordination is produced when two or more subsystems are influenced by the coordinator in each control step (i.e. at the system sampling rate), while sporadic coordination results when this influence appears from time to time at a much lower frequency or completely asynchronously. An example of continuously coordinated system is a positionforce controlled two-arm system [START_REF] Zieliński | Motion generation in the MRROC++ robot programming framework[END_REF].

Thus this classification parallels the classification of control system structures. A system composed of a set of separately controlled robots is treated as uncoordinated. A system composed of many robots interacting externally (thus infrequently) is treated as sporadically coordinated. Such systems can have a supervisor, but this is not a necessity. Sporadic coordination can arise also when several effectors of a single robot are controlled by their own control subsystems, coordinated from time to time by a supervisor. Finally, a single-robot system composed of many effectors interacting internally (thus frequently) is treated as continuously coordinated by a supervisor.

In control theory both centralised and decentralised systems are considered.

A system is considered to be centralised if there is a single decision entity governing the activities of the whole system, e.g. [START_REF] Simmons | First Results in the Coordination of Heterogeneous Robots for Large-Scale Assembly[END_REF][START_REF] Nakano | Cooperational Control of the Anthropomorphous Manipulator 'MELARM[END_REF][START_REF] Zhu | Control of two industrial manipulators rigidly holding an egg[END_REF]. On the contrary, decisions in a decentralised system are made in its subsystems and the overall activity of the system either emerges as a result, e.g. [START_REF] Rubenstein | Programmable self-assembly in a thousand-robot swarm[END_REF] or the subsystems directly communicate to establish the resultant behaviour, e.g. [START_REF] Matarić | Issues and approaches in the design of collective autonomous agents[END_REF][START_REF] Parker | ALLIANCE: An architecture for fault tolerant multirobot cooperation[END_REF]. This implies that centralised systems are always coordinated (continuously or sporadically), while decentralised systems are either uncoordinated or sporadically coordinated.

If the system is coordinated its subsystems communicate. Communication can be realized in several forms: as peer-to-peer (one-to-one) or as broadcast (one-to-many) as well as many-to-many. In the case of peer-to-peer, a one-toone direct communication link must exist, e.g [START_REF] Matarić | Cooperative multi-robot boxpushing[END_REF] presents a system composed of robots using one-to-one communication approach in multi-robot box-pushing task. In the broadcast form the message producer dispatches it to many receivers (e.g. [START_REF] Matarić | Issues and approaches in the design of collective autonomous agents[END_REF][START_REF] Parker | Alliance: an architecture for fault tolerant, cooperative control of heterogeneous mobile robots, in: Intelligent Robots and Systems '94[END_REF][START_REF] Parker | ALLIANCE: An architecture for fault tolerant multirobot cooperation[END_REF][START_REF] Asama | Development of task assignment system using communication for multiple autonomous robots[END_REF][START_REF] Huntsberger | Distributed control of multi-robot systems engaged in tightly coupled tasks[END_REF]), who either utilize those messages at their will or may even ignore them. The many-to-many communication can be organized as e.g. a blackboard system [START_REF] Hayes-Roth | A blackboard architecture for control[END_REF] or by stigmergy [START_REF] Bonabeau | Swarm Intelligence: From Natural to Artificial Systems[END_REF] -this is an indirect form of communication. In the case of stigmergy the robots can sense the changes in the environment or observe of the actions of other robots (from the point of view of the observer in this case other robots are treated as part of its environment). For example [START_REF] Rubenstein | Programmable self-assembly in a thousand-robot swarm[END_REF] describes robots communicating using infrared sensors, [START_REF] Trianni | Cooperative hole avoidance in a swarmbot[END_REF] presents robots communicating using traction forces, [START_REF] Dudek | Experiments in sensing and communication for robot convoy navigation[END_REF] describes robots communicating indirectly using on-board visual sensing, while [START_REF] Chen | Occlusion-based cooperative transport with a swarm of miniature mobile robots[END_REF] reports on the system composed of 20 e-puck robots pushing three different objects based on inputs from infrared proximity sensors.

Robotic systems (single-or multi-robot) can be also looked at from the point of view of the interaction between the effectors of the robots. An external observer watching how the tasks are realized by one or several robots can expresses the view on the type of cooperation. As the observer perceives the robots only by looking at their effectors those devices are at the center of attention. Cooper-ation results from two or more effectors doing work together. They can act independently of each other, cooperate loosely or tightly. Tight cooperation results when the effectors are coupled directly or indirectly (through an object, possibly even a virtual object), e.g. two manipulators transferring a single rigid object together over a specified trajectory [START_REF] Simmons | First Results in the Coordination of Heterogeneous Robots for Large-Scale Assembly[END_REF][START_REF] Nakano | Cooperational Control of the Anthropomorphous Manipulator 'MELARM[END_REF], object-pushing task [START_REF] Matarić | Cooperative multi-robot boxpushing[END_REF][START_REF] Chen | Occlusion-based cooperative transport with a swarm of miniature mobile robots[END_REF][START_REF] Zieliński | Stigmergic cooperation of autonomous robots[END_REF], two rovers carrying a beam fixed to them [START_REF] Huntsberger | Distributed control of multi-robot systems engaged in tightly coupled tasks[END_REF], several quadrocopters flying in a specified formation [START_REF] Whitzer | In-flight formation control for a team of fixed-wing aerial vehicles[END_REF][START_REF] Tron | Vision-based formation control of aerial vehicles[END_REF][START_REF] Turpin | Trajectory design and control for aggressive formation flight with quadrotors[END_REF]. Obviously transfer of a flexible o flimsy object also fall into that category, e.g. a two-handed robot folding a towel [START_REF] Maitin-Shepard | Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding[END_REF]. Sporadic cooperation appears when the effectors work together from time to time in a common space (e.g. transfer an object from one hand to the other). Independent work of the effectors is equivalent to non-cooperation or independent operation.

Coordination and cooperation are distinct from each other. The former is a white box view [START_REF] Brugali | Trends in robotic software frameworks[END_REF], where the observer knows what is the internal structure of the control system, while the latter is based on a black box approach, where only the external actions of the effectors can be observed. Thus the control system perspective is associated with coordination and the interaction between the effectors is associated with cooperation. Two uncoordinated robots can cooperate, e.g. transferring a common object using stigmergy. If the two control systems agree as to the undertaken actions through direct communication sporadic coordination results. The same is the result of a supervisor exerting influence over the two control systems from time to time. On the other extreme there are coordinated robots that do not cooperate, e.g. one executing welding of two plates and the other polishing a cylinder, however both having a single control system -rather a rare situation. Coordinated control systems producing tight cooperation are employed for e.g. two-handed manipulation of a single rigid object [START_REF] Zieliński | Rubik's cube as a benchmark validating MRROC++ as an implementation tool for service robot control systems[END_REF]. Uncoordinated non-cooperating robots are most abundant (e.g. separate assembly cells in a factory).

The control system is responsible for the activities of the system, regardless whether it is single-or multi-robot. Those activities are undertaken to execute the task. It can be quite simple or very complex with many facets -this is not relevant here. What is important is at what stage and by whom the task has been formulated. The task can be embedded in the controller at its design stage and thus becomes its internal imperative to act. On the other extreme the controller can have an embedded interpreter of a specialized language and by using this language the user conveys the task to the controller -this is employed in the case of industrial robots. Thus the internal imperative is created at runtime (from the point of view of the controller) and, moreover, by the user not the designer. Obviously in-between those extremes exist intermediate solutions, where the designer and the user coproduce the imperative to act -the designer produces a parametrized system and the values (numeric or symbolic) are delivered by the user. Regardless of those cases the structure of the controller is fixed by its designer -it does not change at runtime. The interpreter or the variables holding the parameters had been delivered by the system designer.

The above mentioned abundance of classification criteria causes that there is a plethora of differing control system architectures. However they have one common feature. Their architectural structure is set at design-time. Such systems have an invariant architectural structure at runtime. The system presented in this paper is a single-robot multi-effector one. Its subsystems (agents) are sporadically coordinated. It uses direct communication of peer-to-peer type. Due to its variability it can be treated as decentralized, however at any one time there is a single decision entity governing the activities of the whole system. Thus at any particular instant of time one can point out a single decision unit, but at different instants those decision units will change.

Nevertheless variable structure robot control systems were produced in the past. Those could acquire new modules or dispose them, but the supervisor was always the same, e.g. [START_REF] Ishiguro | A robot architecture based on situated modules[END_REF]. In this case although the modules were added or deleted while the system was running, this was done by the system developer, not by the system itself.

The benefits of using dynamic variability of robot control software was also recognised in [START_REF] Brugali | Dynamic variability meets robotics[END_REF]. Due to the necessity of optimizing the use of robot resources or adapting the robot behavior to changing environmental context the capability to add or remove software modules was treated as an asset. Dynamic variability stems from the concept of dynamic software product lines [START_REF] Bosch | Dynamic variability in software-intensive embedded system families[END_REF] addressing the problem of designing runtime configurable software systems. This concept is being extended to context variability to produce context-aware and self-adaptive systems. The non-context features of the robot, such as its hardware structure, can be taken care of at software design time. Task context features such as accommodation to, e.g. type of carried load, must be addressed at runtime. Thus adding or removing (exchanging) software modules was proposed to deal with such contextual changes. However the supervision of the system did not switch.

On the sideline of this discussion, to avoid misunderstandings, it should be noted that in control theory there exists the term: variable structure control (VSC), e.g. sliding mode control [START_REF] Hung | Variable structure control: A survey[END_REF][START_REF] Bartoszewicz | Remarks on 'discrete-time variable structure control systems[END_REF]. This has nothing in common with variable structure control architectures discussed here. VSC refers to switchable control law, which changes the structure of the mathematical model of the system. This mathematical model is expressed in terms of differential equations whose terms differ in different areas of the system state space. In those systems the architecture remains fixed. In the case presented here the focus is on variable structure of the system architecture not the variable structure of the control law.

The possible variability of the control law is embedded in low-level hardware drivers.

Moreover, there is still another popular dimension in the subdivision of robot control systems that should be mentioned as it is customary to deal with it whenever one writes about robot control system structures, although it is not directly relevant to the treated subject. It pertains to how the robot makes decisions as to the action that should be undertaken. It is either based on the world model and symbolic representation used to formulate the plan of actions or on direct coupling of sensoric data obtained from the environment to the effector. In the former case the system layers where allotted distinct functional tasks: perception, world-modelling, planning action execution -an architecture used for instance by the Shakey robot designed in the 1960s [START_REF] Nilsson | Shakey the robot[END_REF]. In the latter case usually the coupling is implemented by a layered network of computational elements acting concurrently and producing partial decisions which eventually undergo arbitration to produce the desired behavior [START_REF] Brooks | A robust layered control system for a mobile robot[END_REF][START_REF] Brooks | Intelligence without representation[END_REF]. The first approach was favoured by classical artificial intelligence and was initially named sensemodel-plan-act (SMPA) [START_REF] Brooks | Intelligence without reason[END_REF] and later more briefly sense-plan-act (SPA) [START_REF] Kortenkamp | Robotic systems architectures and programming[END_REF].

The second approach was termed as the behavioural approach, because the partial decisions produced by the distinct layers of the control subsystem suggested specific behaviours. The structure of the SPA system assumed separate modules devoted to perception, association of percepts to symbolic representations in the world model, decision making relying on planning and finally execution of the thus elaborated action. The modules primarily processed the data sequentially, however feedback loops existed. In the behavioural case each layer of the structure was connected both to sensors and actuators, thus it produced a suggestion of a certain behaviour. Lower layers had less competence, so the suggested behaviours were more rudimentary. Upper layers could inhibit the outputs of the lower layer components or substitute their own suggestions for theirs, thus improve upon their actions. Nevertheless, subsequently it turned out that both approaches, i.e. SPA and behavioural, are compatible and can be merged into a hybrid form [START_REF] Arkin | Behavior-Based Robotics[END_REF] with the lower control layer being reactive (behavioural) and the upper one being deliberative. It should be noticed that the problem how the decisions are formed strongly depends on the task to be realized. The architectural structure of the system should not constrain the designer of the system to implement the task in a way most suitable to it. In this paper this criterion will not be used, as it presents a structure where the task is encoded in a user delivered Dynamic Agent. The proposed architectural structure does not constrain the designer to use any of the above described approaches.

CONTROL SUBSYSTEM

Agents

The RAPP system architecture is described in terms of agents. It uses two kinds of agents: embodied agents (fig. 1) and computational agents. The former are composed of: a control subsystem, one or more virtual effectors and zero or more virtual receptors. The control subsystem is responsible for the realisation of the task of the agent, while the virtual entities serve as device drivers for real effectors and real receptors -the hardware constituting the agent. The controlled hardware constitutes the body of the agent, hence an embodied agent.

Computational agents, which do not posses effectors and receptors contain just the control subsystem. Thus embodied agents are a generalisation of computational agents. So the activities of both kinds of agents are described in the same way. One robot can be controlled in conjunction by several agents.

The control subsystem as well as the virtual effectors and receptors are specified in terms of FSM (Finite State Machine) defining sequences of behaviours [START_REF] Kornuta | Robot control system design exemplified by multicamera visual servoing[END_REF][START_REF] Zieliński | A systematic method of designing control systems for service and field robots[END_REF][START_REF] Zieliński | Diagnostic requirements in multi-robot systems[END_REF]. The behaviours themselves are defined by transition functions and terminal conditions. Each transition function takes as arguments the input buffers to the subsystem and the contents of its internal memory and produces the values of output buffers and new values of the memory. This is done iteratively at subsystem sampling rate until the terminal condition is fulfilled.

Terminal condition is a predicate taking as arguments the contents of input buffers and the memory. Iterations of the same transition function until the satisfaction of the terminal condition are called a behaviour. The FSM switching its internal state switches the behaviours associated with those states, i.e.

switches different transition functions.

RAPP system structure

As the structure of a RAPP system changes over time, it is best presented as a sequence of system structures evolving in stages. The system is composed of a RAPP Platform and one or more robots. We assume that the RAPP Platform exists from the onset and robots know how to connect to it, but at initiation the RAPP Platform and the robot are separate entities (fig. 2a). To simplify the presentation a one robot structure will be considered here. Each robot is delivered with a Core Agent a core controlling it, thus at first it has to establish communication with the RAPP Platform (fig. 2b).

Functions of the agents

The article [START_REF] Dudek | Nao Robot Navigation System Structure Development in an Agent-Based Architecture of the RAPP Platform[END_REF] presents the distribution of system functions between particular RAPP agents and theirs subsystems. Next, the functions of the agents are presented, starting with the agents located in the robot controller and then those located in the RAPP Platform. Each robot included in the RAPP system contains a specific Core Agent a core delivered by its manufacturer. This agent is responsible for communication with the user, downloading the Dynamic Agent a dyn from the RAPP Store a store and providing to the Dynamic Agent the interface to the robot's hardware -directly through its control subsystem and indirectly through its virtual effectors and receptors that are specific to particular robots. The activities of the Core Agent a core are described by the FSM in fig. 5. In reality this are the activities of the control subsystem c core of the Core Agent a core , but as in this paper, besides this subsection, the other subsystems will not be addressed, the agent and its control subsystem will not be distinguished. Computational agents contain only their control subsystems, thus this distinction in that case is not necessary as well. After initialization a core registers itself with the RAPP Platform by connecting to the Platform Agent a plat and the RAPP Store a store and subsequently starts to listen to the user's commands, which can be expressed in one of the two forms: as an isolated keyword or a long command. In each of those cases interpretation is different. The isolated keywords are known to a core and so it decides which Dynamic Agent should be downloaded from the store in response.

Core Agent

The long commands are only recorded by a core and then sent for interpretation to a plat . The Platform Agent a plat decides which Dynamic Agent should be downloaded by a core . Successfully interpreted user's command results in downloading and activating the adequate Dynamic Agent a dyn . Herein, for brevity, different Dynamic Agents are not distinguished by an additional subscript. If interpretation of the command fails, a core returns to listening to the user. After a dyn has been activated, a core waits for its commands and subsequently executes them. Once a dyn has completed its task it notifies a core about that. This results in a core destroying a dyn . Subsequently a core goes back to listening to the user's commands.

The Core Agent a core acts as a service provider for the Dynamic Agent a dyn . Whenever a dyn needs the resources of the robot, i.e. its effectors or receptors, to carry out the task it must request a service from a core , which waits for commands in the FSM state represented by the graph node s 8 (fig. 5). Each such service is represented as a sub-FSM symbolised by the double encircled node s 9 . It should be noted that all real-time critical services are provided by fig. 5 in its state s 9 , so the delays introduced by the communication with a dyn and a cloud may slow down the activities of the system as a whole, but will not affect the real-time performance of the robot hardware.

Dynamic Agent

The Dynamic Agent a dyn is responsible for the execution of the task commanded by the user and is dynamically initiated on the robot with the start of the task. The graph of the FSM representing the required sequence of behaviours depends on this task. The Dynamic Agent communicates with other agents using a standardized API. The Core Agent has to be organized in such a way as to execute all of the API functions invoked by all the Dynamic Agents, hence the detailed implementation of the API functions in the Core Agent as well as the robot hardware do not affect the task formulation in the Dynamic Agent.

The Dynamic Agent FSM is designed by the RApp provider. An exemplary Dynamic Agent a dyn responsible for the execution of a hazard detection task will be presented in section 7. Its graph is presented in fig. 16 in that section.

Platform Agent

The Platform Agent a plat presents its computational capabilities to the Core Agent a core and the Dynamic Agent a dyn as a set of services that can be grouped into few general subsets. Communication services currently enable the robot to perform automated speech recognition (using either Sphinx [START_REF] Tsardoulias | An automatic speech detection architecture for social robot oral interaction[END_REF] or Google libraries), speech synthesis as well as sending and receiving e-mail messages [START_REF] Figat | Distributed, reconfigurable architecture for robot companions exemplified by a voice-mail application[END_REF].

Those services can be used directly by the Core Agent to recognize the command produced by the user, when the robot is not capable of recognising the command using only the on-board skills. Another large subset of services is responsible for image processing and object detection. To recognize people, there are human detection as well as face detection and recognition services. Other services enable scene recognition and object detection. The light checking and door-opening estimator makes it possible to create applications automatically assuring user safety. The navigation subset contains map management services and enables global localization (initially based on QR codes and currently doing without them). Using global obstacle map, the path between two points can be planned.

A set of information services (geolocalization, news and weather) and ontology (Knowrob) supplements the platform capabilities. Fig. 6 presents the general structure of the FSM governing the agent a plat . s 1 Wait for service request from a dyn or a core s 2 Provide the service requested by a dyn or a core

Cloud Agent

A Cloud Agent must be compatible with its respective Dynamic Agent -it is created by the RApp developer as a part of a particular application. It is meant to be available only to this one application, and its lifetime is similar to the lifetime of the Dynamic Agent, i.e. when the Dynamic Agent is launched, the corresponding Cloud Agent is launched too, and both terminate their existence at the same time.

The Cloud Agent is responsible for the execution of tasks, that are too computationally expensive to run them on the robot controller, yet not general enough to be implemented as platform services. The Cloud Agent can be also used as a macro-service, calling multiple platform services and returning the final result to the Dynamic Agent, allowing it to run other tasks in parallel without time-consuming back-and-forth communication and dealing with intermediate service results.

RAPP Store

The RAPP Store caters to: robot owners, RApp developers and the Core Agents controlling the robots. Every new robot owner registers with a store .

The Core Agents register with a store and a plat and thus gain access to the RAPP infrastructure. From the point of view of the Core Agent a core the most important service provided by a store is the ability to download RApps, i.e. the Dynamic Agents a dyn required by the user. The RAPP Store a store is also responsible for providing services to the RApp developers for building and submitting RApps. Its obvious responsibility is storing RApps. The RApps are hosted in the form of precompiled binaries of Dynamic Agents represented by a dyn .

Implementation

Implementation of the RAPP infrastructure follows the agent-structure, depicted in fig. 4b. As the RAPP system is rather complex, an in-depth description of the implementation of every single component is beyond the scope of this paper, for obvious reasons -rather the general idea and the main concepts will be explained. This section is organized in top-down manner, describing top-level modules and communication between them first and then going deeper into each submodule and service.

Communication between agents

Figure 7 presents the same blocks as fig. 4b, however from a different perspective, focusing on the communication between the top-level entities, i.e. agents.

Apart from the Dynamic Agent, every other module exposes its services (or, in general, capabilities) to the public. To make possible the use of them by the others, every agent has an API prepared for that purpose.

RAPP Store

The RAPP Store exposes just a few services to be used mostly by the Core Agent. Those are available using the REST API (HTTP request-response). It enables the listing of available applications, searching for a specific one, gathering information about the selected application and downloading a compiled package to run as a Dynamic Agent. Internal structure of the Store is presented in fig. 8. The whole interaction between the developers and the Store is possible using web interface.

The

RAPP Platform

The RAPP Platform [75] provides its services as web services, implemented using HOP [START_REF] Serrano | Hop: a language for programming the web 2[END_REF][START_REF] Serrano | Multitier programming in Hop -a first step toward programming 21st-century applications[END_REF]. Some robots such as NAO can recognise only a limited set of commands. Others, such as the Elektron mobile robot do not have the recognition capability at all. The client uses the Platform API (available for Python, The Platform API, among others, is responsible for authentication -each call to the platform is supplemented with a user token. Based on it the user status is ascertained (e.g. it is verified whether the selected service is available for the given user or not) as well as adequate data storage is selected (e.g. each user has a personal set of face images to be recognised, maps for navigation etc.). If the service is rudimentary, such as file upload or database query, it can be implemented directly in HOP. More complex algorithms need the RAPP Improvement Center (RIC), which uses ROS [START_REF] Quigley | ROS: an open-source Robot Operating System[END_REF][START_REF] Cousins | Ros on the pr2 [ros topics[END_REF]. Each node provides one or more services, which are called from HOP using rosbridge websockets server [START_REF] Crick | Rosbridge: ROS for Non-ROS Users[END_REF].

ROS enables many possibilities of node implementation. The majority of the RAPP system modules is implemented using either C++ or Python. Internally, each ROS node runs multiple processing threads, so that it can process calls from many clients simultaneously. Taking into account how ROS threads are handled, it becomes apparent that ROS nodes that are purely functional (i.e. do not posses an internal state or a back-end procedure) have no problems with simultaneous calls. On the other hand, nodes such as the Knowrob Wrapper or the Sphinx4 ASR system, which depend on the deployment of other packages, need special handling. In the case of RAPP Platform, it was decided that only the Sphinx4 ASR ROS node is in need of such service handling, since this will be the most frequently invoked one. The assumed solution involves the creation of several back-end Sphinx4 processes, along with a handler, which accepts the requests and decides which process should serve them. At any time, the handler monitors the processes to keep track which of them are occupied and what are their current configurations. Thus if, for example, a speech recognition invocation occurs with a specific configuration, the pool of unoccupied processes is tested to find out whether any of them is already configured as requested. If this is the case, the handler proceeds directly to the speech recognition process, or if not, the handler selects a random unoccupied process and performs both configuration and recognition. This approach was chosen, because the configuration process is quite time consuming and requires significant resources, thus it should be avoided whenever possible.

The RAPP Platform contains the RAPP database which can be accessed by:

• the RAPP Store, to keep track of the user accounts and the submitted/downloaded applications and

• the RAPP Improvement Centre to provide machine learning/statistical data gathering and data acquisition -this is done via a ROS MySQL wrapper, providing interfaces to all the nodes in need of stored data. As it has been said in section 5.1, the Core Agents implementation is dependent on the hardware it controls. On the other hand, the set of functions available for the Dynamic Agents to call must be known in advance and, as much as possible, independent on the particular robot. In social robotics, in particular in tasks for which RAPP is designed, those functions can be divided into three main groups: communication, vision and navigation skills.

Communication is required by the Core Agents main automaton -interaction with the user is done using spoken commands and synthesized replies. Thus, at least two skills should be present: say -to synthesize a sentence; and word spotting -listen to and recognize words. Other useful functions are capturing and playing back audio files.

As the sight is one of the most important human senses, and one giving the most information, also robots vision services plays a key role in whole system design. Main function in this group is responsible for capturing pictures, and a few helper functions should also be created for changing camera parameters, reading current settings and calibration etc. Robot should also be able to look at given point in space, to look for specified object for example.

Last, but not least, in order to cooperate with people and to be able to move around in a home, the robot should possess some kind of navigation capabilities. Those robots are very different in their capabilities. NAO is a humanoidthis implies that it can execute holonomic motion, i.e. walk in any direction; while Elektron having wheels and a differential drive is non-holonomic. NAO is equipped with two medium quality cameras on a moving head, while Elektron has one high quality moving camera and two fixed Kinect sensors. Table 1 compares the relevant components of each robot. First, control subsystem, managing the FSM (fig. 5) is almost independent of a particular robot. The control subsystem is written in the C++ programming language, as a ROS node, thus any robot which is ROS-enabled and has basic abilities record and playback sound. If the robot has no built-in capabilities to recognize spoken text and transform text to speech (as is the case of the Elektron robot), the Core Agent can send the recording to the Platform Agent for recognition.

The FSM, after successful recognition of the user command, downloads a compressed archived file with the selected RApp and unpacks it. As the Dynamic Agent can be developed in one of three possible languages: C++, Underlying services, provided by the two aforementioned robots, are implemented as ROS nodes. In the case of the NAO robot the virtual effectors are: e core,body that is responsible for the motion of all of its limbs and the head, and e core,ls that governs the loudspeakers and executes text to speech transformation. The NAO virtual receptors are: r core,cam that is responsible for acquiring images by the camera, r core,sonar that detects obstacles using ultrasonic sensors, r core,touch that detects obstacles using micro-switches located in the feet of the robot, and r core,mic that detects sound using microphones and recognizes words.

All those virtual effectors and receptors are commanded by the control subsystem c core (fig. 13). Those subsystems are implemented in Python. The virtual effectors and receptors contact the hardware via the NAOqi client library (provided by the robot manufacturer), thus this library is treated as a part of real effectors and receptors. When requested, the Platform Agent first checks in the Store whether the desired package is available, then downloads it and builds using scripts provided with the package. After successful build, the created application is deployed on a new Docker container (lightweight software virtualization [START_REF] Merkel | Docker: lightweight linux containers for consistent development and deployment[END_REF]) instance and the identifier of the created Cloud Agent is returned to the calling Dynamic Agent [START_REF] Thallas | Relieving robots from their burdens: The Cloud Agent concept[END_REF]. This identifier is then used when calling services provided by the Cloud Agent. Those services form the second part of the Cloud Agent communication mechanism, and are strictly dependent on the application, i.e, RApp.

Experiments

Forgetfulness is a typical problem for elderly people suffering from MCI.

For instance, leaving the house they tend to forget shutting windows, balcony

doors, leaving open refrigerator doors or lights switched on. Detection of such hazards can be handled by equipping the home with a multitude of sensors, however elderly people usually dislike any modifications to their homes and subsequent maintenance of the resulting system. Thus a different remedy can be recommended. A robot can detect all potential risks by exploring the home.

In our experiments two different robotic platforms have been employed to carry out this task, the NAO humanoid robot and Elektron wheeled mobile robot (fig. 15). Besides the locomotion mechanism these robots differ in their onboard sensory systems. Once the robot global pose is computed, the Dynamic Agent sends this pose and the requested hazard pose to the RAPP Platform agent to gather a collision-free path to the pose suitable to analyse the particular hazard. This is done by the behaviours associated with states s 3 and s 4 .

Once the path is obtained its execution commences. The robot moves to the way-points located along the path; however not in each of them it has to look for hazards. At each way-point the robot has to self-localize. All this is done by a subFSM symbolised by s 5 . The performed experiments show that the robot localizes itself with good accuracy (for the NAO robot the localization error is ±10 cm and for the Elektron is ±7 cm. This enables better tracking of the desired path. The robot follows the desired path with adequate accuracy.

The maximum path tracking error is 15 cm for the Elektron robot and 20 cm for the NAO. This is a satisfactory result for such a simple robot. In general, path execution strongly depends on the robot effectors and sensors, and its Core Agent abilities.

Hazard detection depends on the type of hazard and on the object it is associated with. We use monocular vision to collect hazard information. In locations where potential hazards may occur, the Dynamic Agent requests the Core Agent to orient the robot's camera to point at the possible hazard point, e.g. lamp, door. In the case of open-doors detection the state of the doors is based on image analysis process that starts with looking for all sufficiently long linear segments, that are further categorized into vertical and horizontal ones (white or gray as shown in fig. 17). From the horizontal lines, two are selected, that form wall/floor and door/floor intersections. Those are selected as the longest lines with an inclination closest to the horizontal line. The algorithm is prepared in such a way, that the found lines are always localized on opposite sides of the picture (left and right). Finally the door-opening angle estimation is made by simply calculating the angle between the two found lines (marked in black in the pictures in fig. 17).

(a) (b) In the case of searching for the lights that had not been turned off, the robot moves its camera to look at the place that the lamp is located in. Then an image is captured and image processing is performed to detect an intensive light. Light status is checked by analyzing the picture taken with low exposure, while looking directly at the light source (as presented in fig. 18).

(a) (b) Size of the rectangles is selected in such a way, that the central one is the most important, while surroundings form a rough approximation of a circle. The result is normalized to the range [0..100], where 0 means that the light is probably turned off and 100 maps to the light turned on without doubt.

Next the Core Agent is notified about the detected hazards, so that it can report them to the user -the behaviour associated with state s 6 is responsible for that. Finally the Dynamic Agent informs the Core Agent that the task is done, so that it can destroy a dyn -this is done by the behaviour associated with s 7 .

Conclusions

Besides the hazard detection task, which was tested both on the NAO and Elektron robots, the RAPP system has been tested on several other benchmark examples, e.g.: voice e-mail via NAO [START_REF] Figat | Distributed, reconfigurable architecture for robot companions exemplified by a voice-mail application[END_REF], [START_REF] Szlenk | Reconfigurable Agent Architecture for Robots Utilising Cloud Computing[END_REF], NAO the weather reporter, news explorer via NAO, playing cognitive games with NAO. The multiplicity of both the robots used and the tasks executed convince that the assumed control architecture is fully functional. The proposed architecture solves the problem of high computational requirements versus limited resources.

Figure 1 :

 1 Figure 1: Internal structure of an embodied agent

Figure 2 :

 2 Figure 2: (a) RAPP system initial structure, (b) Connection of the Core Agent acore to the RAPP Platform a plat

Figure 3 :

 3 Figure 3: (a) Download and creation of the Dynamic Agent a dyn , (b) Connection of the Dynamic Agent a dyn to the Platform Agent a plat

Figure 4 :

 4 Figure 4: (a) Creation of the Cloud Agent a cloud , (b) Connection of the Dynamic Agent a dyn to the Cloud Agent a cloud

s 1 Initialization s 2 6 7 8 9 11 Unregister from a plat and a store s 12 Figure 5 :

 2678911125 Figure 5: Graph of the FSM governing the behaviour of the Core Agent acore

Figure 6 :

 6 Figure 6: Graph of the FSM governing the behaviour of the Platform Agent a plat

Figure 8 :

 8 Figure 8: Internal structure of the RAPP Store

Figure 9 :

 9 Figure 9: Internal structure of the RAPP Platform Agent

Figure 10 :

 10 Figure 10: NAO robot: description of relevant components

Figure 11 :

 11 Figure 11: Elektron robot: description of relevant components

Figure 12 :

 12 Figure 12: Internal structure of the RAPP Core and Dynamic Agents

Figure 14 :

 14 Figure 14: Core Agent of Elektron robot

Figure 15 : 1 2 -localization s 3 4 5 6 7

 151234567 Figure 15: Robots performing hazard detection: (a) NAO humanoid robot, (b) six-wheel Elektron robot

Figure 16 :

 16 Figure 16: Graph of the FSM governing the behaviour of the Dynamic Agent a dyn realising the hazard detection task

Figure 17 :

 17 Figure 17: Sample images for door-opening angle estimation

Figure 18 :

 18 Figure 18: Sample images for light checking

  RAPP Store is responsible for the whole process of application prepa-

	Store Agent	Platform Agent		Cloud Agents
	Application		Request package	Cloud Agent	Ecosystem
	repositories	RAPP Improvement	Manager	Deploy Cloud Agent
			Center (RIC)
	server Store WEB	ROSbridge Service Platform WEB server req/resp DB	Launch control	Cloud Agent #1 Cloud Agent #1 Cloud Agent #1 Cloud Agent #1
	Store REST API		Platform WEB API
					Call Cloud Agent
	Download		Call Platform	Launch	services
	application		services	Cloud Agent	Dynamic Agent
	Core Agent
	Core Robot services	Robot API	Call Core Agent services	Rapp Client Libraries (C++/JS/Python)
	Robot FSM	Launch Dynamic Agent
	ration. Every developer can create many git repositories, each containing the

Figure 7: Overall architecture of the RAPP infrastructure

source codes of one RApp. When the code is ready, the programmer can release his/her application for use by the selected robotic The RAPP Store is responsible for RApp compilation (if necessary) and packaging. There is a separate virtual machine for each available robot. It is used as a build server.

Platform Agent DB Platform WEB server RAPP Improvement Center

				Cloud Agents infr.	Store
					Cloud Agent
	ROS nodes (excerpt)			Manager
	/path_planning	/hazard_detection	/authentication_manager
	Services:	Services:		Services:
	/plan_path_2d	/light_check, /door_check	/add_user, /login, /authenticate_token
		Service req/resp (TCPROS)
				ROSbridge
			Service req/resp (websockets)
	Authent. worker	Hazard det. worker	Navigation worker	Cloud control
	authenticate	light_check	door_check	upload_map path_plan_2d	launch_cloud_agent
	HOP Web Broker HTTP req/resp	exec invoke
	Platform WEB API
		Clients

Table 1 :

 1 Comparison of components of the robots

	Function		NAO		Elektron
	Say	Internal sentence synthesizer	N/A
	Word spotting	Internal ASR service	N/A
	Recording	Four microphones	One microphone
	Playback		Speakers		Speakers
	Capture image	Two RGB cameras	One RGB camera
		1280x960px, wide angle	1280x1024px, narrow angle
					Two Kinect RGB-D sensors
					640x480px, wide angle
	Mobility	Two legs, holonomic movement Differential drive, non holonomic
	Speed		??		Up to 0.25m/s
	Python or JavaScript, the RApp developer has to prepare the deployment script,
	responsible for correct start of the application (in the simplest case just calling
	the executable). This script is initiated by the Core Agent as a new process
	(fork-exec procedure). Task is treated as finished when this process dies -the
	Core Agent monitors its state and detects this event.
		Download			Launch	Cloud
		application		Cloud Agent	Agent
	Core Agent Robot FSM		Launch Dynamic Agent	Dynamic Agent cloud_agent control library (C++/JS/Python)	service invocation
	move_to, take_predefined_pose, say, word_spotting, ... Robot API look_at, take_picture, Core Services	invocation service Robot	(C++/JS/Python) robot_api client library (C++/JS/Python) platform_api client library cloud_agent client library (up to application programmer)	invocation Platform service

  The virtual effectors of the Elektron robot are: e core,base that is responsible for the motion of the mobile base, e core,head controlling the movements of camera supporting head and e core,ls that governs the loudspeakers. Virtual receptors are responsible for gathering images from multiple cameras: r core,rgb from RGB camera, r core,kin_top and r core,kin_bot from Kinect sensors. Laser scanner is managed by r core,laser , and r core,mic records sounds using microphone (fig.14). The Elektron robot software was implemented in C++ as a set of ROS nodes.

	Core Agent (Elektron)
	c core						Robot API
	e core,body	e core,head	e core,ls	r core,kin_top	r core,kin_bot	r core,mic	r core,laser
	Mobile base	Head	Loudspeakers	Upper Kinect	Lower Kinect	Microphone	Laser scanner
	Core Agent (NAO)
	c core						Robot API
	e core,body	e core,ls	r core,cam		r core,sonar	r core,touch	r core,mic
	Body + head Loudspeakers	Upper camera	Lower camera	Sonar sensors	Touch sensors	Microphone
			Figure 13: Core Agent of NAO robot

Acknowledgement

This work is funded by the EU Commission within the FP7 Collaborative Project RAPP (Grant No 610947). Moreover it is supported by the Polish Ministry for Science and Higher Education scientific research funds for the years


Jean Launay

Fathia Lahmar

Philippe Boisse

Pierre Vacher

STRAIN MEASUREMENT IN TESTS ON FIBRE FABRIC BY IMAGE CORRELATION METHOD

Strain measurements are investigated by an image correlation method, for tensile tests, in order to analyse the non-linearities due to crimp. The interest of the method is the possibility to study the phenomena at different scales and so to contribute to a better understanding of the yarn interactions and of their influence on the fabric mechanical behaviour

INTRODUCTION

Fabrics are made of yarns that are intertwined in regular patterns (plain, twill, satin) and give a flexible surface. Their stiffness is only significant in tension, in the yarn directions. They are very efficient reinforcements for thin composite structures, especially in case of double curved parts for which the unidirectional reinforcements cannot normally be used [START_REF] Billoet | Introduction aux matériaux composites a hautes performances[END_REF][START_REF] Gay | Materiaux Composites[END_REF]. The knowledge of the mechanical properties of the fabrics is necessary both for stretched structure analyses and, when manufacturing composites, for the simulation of forming stages before resin injection (first stage of the R.T.M. process) [START_REF] Boisse | Meso/ macro-mechanical behaviour of textile reinforcements for thin composites[END_REF]. Those mechanical properties are very specific because of the possible motions between the yarns and the fibres within the yarns . In this paper, tensile tests on woven reinforcement are undertaken for fabrics used in R.T.M. composite manufacturing process. The mechanical tensile properties of the fabrics are analysed by a biaxial tensile test, in order to measure the non-linear behaviour due to the change of shape of the undulations (given by weaving) when the fabric is in tension. The strain measurement that are necessary in these tests are delicate. The classical efficient local measurements (extensometers, strain gauges) cannot be used. A local sensor easily perturbs the very weak stiffness, due to the fibrous constitution of the fabric. Some optical techniques using digital photographs have been used for fabric biaxial tensile tests [START_REF] Borr | Détermination du comportement biaxial des tissus de fibres de verre. Mesure optique de la déformation, Etude du comportement des materiaux et des structures[END_REF]. The difficulty of the procedure leads to use of a CCD camera associated to an image correlation method in the present paper. The optical measurement gives the mean value of the tensile strain over the specimen and a field of strains. It permits a check on the homogeneity of the strain field in the active part of the specimen and also analysis of local variation of the strains near the yarns.

2.EXPERIMENTAL ANALYSIS OF FIBRE FABRIC MECHANICAL BEHAVIOUR 2.1.Mechanical behaviour of fibre fabrics

Fabrics are made of woven yarns themselves composed of juxtaposed fibres. The diameter of these fibres is very small (generally 5 to 7 mm for carbon fibres and 5 to 25 mm for glass fibres). Consequently these fibres can only be submitted to a tensile stress in the fibre direction. Because they can slide relatively in case of yarn curvature, the yarn can only be submitted to a tension. The mechanical behaviour of the fabric is thus defined only by the two tensions in warp and weft direction, which depend on the two corresponding deformations. Different approaches can be used to determine the tension-strain curves. A biaxial tensile test is described in the next section. 3D numerical simulations of the elementary pattern can also be used to define these curves [START_REF] Gasser | Mechanical behaviour of dry fabric reinforcement. 3D simulations versus biaxial tests[END_REF].

Biaxial tensile tests

The device shown Fig. 1a is directly set on a classical traction/compression machine [START_REF] Ferron | Dispositif de traction biaxiale DAX2. Document Techmetal[END_REF]. It works via strain ratio in warp and weft directions. This ratio can be fixed at six different values. The crosshead speed is 10 mm/mn. For a non-deformed fabric, the yarn nets are perpendicular to each other, for traditional 2D fabrics. Nevertheless, when a fabric is deformed, the angular variations can be important especially when the fabric is formed on a double curved surface. The biaxial tensile device allows setting the angle between weft and warp yarn directions to a value different from 90°, but this variable option is not used here.

The cross shape of the sample is shown Fig. 1b. The studied part is the woven central square. The ends are not woven. The cross shape is particularly well adapted to biaxial tests on woven materials because they have no in-plane shear stiffness so the stress/strain field can be considered as homogenous at the macroscopic level in the active part of the cross shape specimen. Defining the initial state is a delicate problem. A load criterion to determine the precise start time of the test is not adopted in this analysis because of the very low loads at the beginning of the test. A kinematical criterion is used. It rigorously marks out the beginning of the test with the instant where every yarn of each direction comes simultaneously into the mean plane of the device (before the test begins the fabric is bending under is own weight and has a bowl shape). This moment is characteristic and can be easily identified.

In order to avoid the perturbations due to friction in the mechanical device, load cells are directly set behind the specimen and give total load on the fabric specimen in each direction.

Strain measurements with mechanical extensometers

The definition of tensile curves requires the measurement of the two axial strains in warp and weft directions respectively. The lack (or the very low value) of some stiffness render very difficult or impossible the use of sensors directly placed on the fabric. Mechanical extensometers cannot be clamped directly on the fabric specimen otherwise they modify the local properties. Nevertheless, measures have been performed, by clamping the two extensometers on the rigid extremities of the crossshaped specimen. In order to obtain the strains in the woven part of the specimen, the stretching in the non-woven part must be subtracted. These values are obtained with a tensile test on a yarn sample. In this manner, the tensions can be obtained as function of the strains for different ratios k (k = warp strain/ weft strain). For example, the results are given in Fig. 2 for a carbon weave. It is a 2*2 twill carbon with 0.35 untwisted yarn/mm. It can be seen that the non-linearities due to the change of shape of the yarns are important. Other biaxial tensile curves of different woven reinforcements used in R.T.M. process can be found in [START_REF] Launay | Analyse expérimentale et modèles pour le comportement mécanique biaxial des renforts tissés de composites (Experimental analysis and models to study the biaxial mechanical behaviour of wowen reinforcements in composite material[END_REF].

Even where, measurements made using extensometers are possible, they have major drawbacks:

a) The measurements have to be posttreated, which renders the analysis more complicated. Knowledge of the nonwoven part tensile stiffness is necessary and must be determined very precisely. b) Only a mean strain value is obtained in each direction. This does not permit either a check on the homogeneity of the strains or determination of local strain measurements.

The optical measurements described below aim to avoid these drawbacks, and also to measure directly the strain field.

IMAGE CORRELATION IN TENSILE FABRIC TESTS 3.1. Strain measurement obtained with digital image correlation method

This method is based on numerical treatment of grey scale digital images representing the sample at different deformation states. Each image, obtained with a CCD camera BASSLER A113 P, has a size of 1030*1300 pixels. The first image, representing the undeformed specimen is divided into equal squares whose size can be of 4*4 to 40*40 pixels (Fig. 3). The four corners of each square are then localised on each image of the deformed sample by using a correlation method [START_REF] Vacher | Bidimensional strain measurement using digital images[END_REF]. This position can be obtained with an accuracy better than 1/20 pixel. That gives the displacement field from which the Green-Lagrange strain tensor can be deduced.

No contact with the observed surface is needed. The measurable strain is between 0.01% and more than 100% and thus is favourable for biaxial tests on fibre fabric.

$ $ ! !$ ! " # $ % & ThvÃÈ G h q à I h `h x2!

Biaxial tensile test on fibre fabric at macroscopic level.

A biaxial tensile sample of an unbalanced glass plain weave is used for this experiment. The yarns have an approximate diameter of 1 mm (1.75 yarn/mm in the warp direction and 1.2 yarn/mm in the weft direction). The central part of the sample is a square of 50*50 mm 2 . Paint pulverisation, with commercial products of three different colours, gives a randomised image of this square with distinct points of 0.05 to 0.1mm diameter and a large grey scale.

The upper part of the biaxial tensile device is modified to include the camera. The lens is chosen to observe all the sample central part. The observed yarn width is then about 10 pixels. Some holes of 7*3 pixels size appear between yarns where they cross together. Their colours on the images are uniform and do not disturb the calculation. From these surfaces, obtained for different loads, a mean strain value is deduced and used to draw the load/strain curves. One of these curves, presented on Fig. 5, compares the results obtained in warp direction to those coming from the extensometer measurement. The curves obtained on the same sample, but not at the same time, have a slight gap along the strain axis. The small difference comes from the post-treatment of the results when the extensometers are used.

For this application, image correlation method gives some improvements.Yarn rigidity measurement and subtraction of the strain of the non-woven part are not necessary; the accuracy can thus be increased.

The image correlation results are superior to those from extensometer; they give displacement and deformation full fields, rather than a mean value of one component. That allows checking of he homogeneity of the strains in the active part of the specimen. The measurement process can also be automated to give a sufficient number of points for each load/strain curve.

Fig. 5: Load/strain curve in the warp direction for k =1, obtained with the two methods.

Strain measurement at the yarn size

With suitable lenses, the digital image can be focused on a specific part of the sample to visualise the strain field at different levels [START_REF] Gonzalez | Strain inhomogeneity and discontinuous crack growth in a particulate composite[END_REF]. For fibre fabric, the interaction area between weft and warp yarn has an important part to explain mechanical behaviour. On The Fig. 6a represents the strain component yy on this area (y is the tensile and the weft directions), with small values in warp yarn and higher values in tensile (or weft) direction.

These examples illustrate the ability of the image correlation method to contribute to the knowledge of the fibre fabric behaviour at different levels. It can easily replace extensometers to obtain load/strain curves, and associated with a 3D finite elements simulation of mechanical tests [START_REF] Gasser | Mechanical behaviour of dry fabric reinforcement. 3D simulations versus biaxial tests[END_REF] could contribute to enhance the behaviour model of woven fibre fabrics.

Fig

  Fig. 1a.: Biaxial tensile device Fig. 1b : Scheme of the sample

Fig. 2 .Fig. 3 :

 23 Fig. 2. Tension load versus strain for different ratio k, for 2*2 twill carbon

Squares of 10 *

 10 10 to 40*40 pixels can be used to calculate the displacement field. The displacement vectors on the central part of the sample are represented on Fig. 4a. Such figure gives an information on the quality of the global biaxial effect for k=1.

Fig. 4a :

 4a Fig. 4a: Displacement vectors 4b: Strain surface for e yy *10000

  Fig. 6b, yarn size is about 150 pixels. This is sufficient to measure displacement field at the cross-over of two yarns. The case shown on this figure is a uniaxial tensile test (k=0) in the weft direction. Correlation calculations concern only the extracted area shown on Fig. 6b.

Fig. 6a :

 6a Fig. 6a: Strain component (e yy *10000)


Harold Mouch Ère
email: harold.mouchere@irisa.fr

Sabri Bayoudh
email: sabri.bayoudh@irisa.fr

Eric Anquetil

Laurent Miclet

Synthetic On-line Handwriting Generation by Distortions and Analogy

One of the difficulties to improve on the fly writer-dependent handwriting recognition systems is the lack of data available at the beginning of the adapting phase. In this paper we explore three possible strategies to generate synthetic handwriting characters from few samples of a writer. We explore in this paper both classical image distortions and two original ways to generate on-line handwritten characters: distortions based on specificities of the on-line handwriting and a generation based on analogical proportion. The experimentations show that these three approaches generate different distortions which are complementary. Indeed the combination of them allows the achievement, using only 4 original characters for the learning phase, of a mean of 91.3% of recognition rate for 12 writers.

Introduction

With the increasing use of the pen-based human computer interface, handwriting recognition systems have to become more and more accurate. One way to improve it is to adapt the system to the handwriting style of the current user as done in a previous work [START_REF] Mouchère | Writer style adaptation in on-line handwriting recognizers by a fuzzy mechanism approach : The adapt method[END_REF]. The difficulty is to learn a new handwriting style from few samples.

In this paper we study three possible strategies to generate synthetic handwriting characters. The first one uses classical distortions from the off-line field. The two next ones are from our original contribution: the second one is based on the particularity of on-line handwriting and the third one is based on the use of analogical proportion on handwriting. Contrary to [START_REF] Varga | Template-based synthetic handwriting generation for the training of recognition systems[END_REF] who generate data to learn a writer-independent system, the synthetic handwriting has to respect the writer script. The quality of this generation is evaluated by considering the performance of a writer-dependent simple classifier learned with synthetic characters generated from very few examples of his handwriting style.

We first present in the section 2 the classical image distortions. The section 3 focuses on the on-line handwriting distortions. The section 4 gives definition and properties of analogical proportion and shows how to use it to generate synthetic characters. The section 5 presents experimental results.

Synthesis by Image Distortions: Scaling and Slanting

Using synthetic data to learn a recognition system is mainly used with off-line systems which recognize an image of the character. The generation is done by using image distortion processes. For off-line character recognition, [START_REF] Cano | Training set expansion in handwritten character recognition[END_REF] use distortions: slanting, shrinking, ink erosion and ink dilatation. [START_REF] Simard | Best practice for convolutional neural network applied to visual analysis[END_REF] extend the learning data base using elastic distortions on images to train a neural network. Moreover text lines can be distorted as in [START_REF] Varga | Generation of synthetic data for an hmm-based handwriting recognition system[END_REF] to train a Hidden Markov Model.

In this paper we limit us to scaling and slanting deformations because of the nature of our data. Indeed we deal with isolated on-line characters inputted directly by the user through a pen-based interface. Thus ink transformations, rotation, perspective and line distortions are not interesting in this case. Two random parameters correspond to the scale transformations, α x and α y which are the ratio of the corresponding scales. The slant allows the generation of inclined handwriting. It depends of one random parameter α s which represents the tangent of the slant. A positive value slants the writing to the right and a negative to the left. The Figure 2 shows examples of distorted characters by scaling and slanting.

Synthesis by On-line Distortions

There are few works about using handwriting generation in order to increase on-line learning data. In the on-line field, the handwriting is considered as a parametric function p (t) corresponding to the pen positions. [START_REF] Varga | Template-based synthetic handwriting generation for the training of recognition systems[END_REF] use works about handwriting generation [START_REF] Plamondon | The generation of handwriting with delta-lognormal synergies[END_REF] to increase the size of a learning database to train an off-line sentence writer-independent recognizer. The authors use a unique on-line handwriting model which is distorted and joined to generate images of complete sentences. This approach is closed to our on-line distortion but unusable in our context because we need writer-dependent models. That is why we propose two simple distortions of on-line handwriting: Speed Variation and Curvature Modification

The aim of Speed Variation distortion is to modify the size of vertical and horizontal parts of the stroke, as shown by Figure 2, depending of a random parameter α v . Indeed these straight parts of the writing can vary without changing the handwriting style. For this we modify the speed V (t) = x (t+1)x (t) , y (t+1)y (t) depending of its direction. If this vector is near one of the axes then it is increased or decreased by the ratio α v . The new synthetic handwriting is defined by p (t) :

p (t) = p (t-1) + β * V (t-1) , with β = 1 if arg( V (t-1) ) π 2 ∈ [ π 8 , 3π 8 ], α v else.
(1)

The Curvature Modification distortion modifies the curvature of the writing as shown by Figure 2. It allows closing or opening the loops of handwriting. The curvature modification uses a random parameter α c . The curvature at the point p (t-1) is defined by the angle θ(t-1) = (p (t-2) , p (t-1) , p (t) ) in ]π, π]. In order to keep the structure of the character, we do not modify the straight lines and cusps. The equation 2 gives the position of the point p (t) depending of the two previous points and of the original curvature θ(t-1) modified by α c . The maximum angular modification is for θ(t-1) = π 2 .

(p

(t-2) , p (t-1) , p (t) ) = θ(t-1) -α c * 4 * θ(t-1) π * (1 - θ(t-1) π ).
(2)

Sequence Generation by Analogical Proportion

Analogy is a way of reasoning which has been studied throughout the history of philosophy and has been widely used in Artificial Intelligence and Linguistics [START_REF] Lepage | Solving analogies on words: an algorithm[END_REF]. We are interested here in a special case of analogy: the Analogical Proportion (AP) between four objects a, b, c and d in the same universe. Having four objects in AP is usually expressed as follows: "a is to b as c is to d". Depending on what are the objects, AP can have very different interpretations.When one of the objects is unknown, as in "wolf is to leaf as wolves is to x", finding a satisfying x is called solving an analogical equation. x would be leaves if we consider either a semantic interpretation of analogy as well as if the interpretation is on the combinatorics of the sequences of letters. More formally, an Analogical Proportion (AP) on a set X is a subset of X 4 . An element of AP writes a : b :: c : d , and reads: "a is to b as c is to d". As defined by [START_REF] Lepage | Solving analogies on words: an algorithm[END_REF]

Analogical Dissimilarity Between Four Objects

When four objects in X are not in AP, it may be interesting to introduce a quantity to measure how far they are from being in AP. This measure, called Analogical Dissimilarity (AD), has been proposed in [START_REF] Bayoudh | Learning by analogy : a classification rule for binary and nominal data[END_REF]. It has been constructed to verify properties coherent with the underlying AP. For example, if the set X is R m , we can define an AP by a : b ::

c : d ⇔ a + d = b + c and a coherent AD is such that: 1. ∀u, v, w, x ∈ R m , AD(u, v, w, x) = 0 ⇔ u : v :: w : x 2. ∀u, v, w, x ∈ R m , AD(u, v, w, x) = AD(w, x, u, v) = AD(v, u, x, w) 3. ∀u, v, w, x, z, t ∈ R m , AD(u, v, z, t) ≤ AD(u, v, w, x) + AD(w, x, z, t) 4. In general, ∀u, v, w, x ∈ R m : AD(u, v, w, x) = AD(v, u, w, x)
Therefore, when defining AD(a, b, c, d) = δ(a + d, b + c), where δ is a distance in R m , it is easy to prove that the four properties above still hold true.

Analogical Dissimilarity Between Four Sequences

Let us define a sequence as an ordered set of objects, these objects being for example vectors in R m , or more generally elements of an alphabet Σ. The set of sequences on Σ is denoted Σ . u = 9 9 9 9 9 9 9 E 1 2 4 L 6 9 9 9 v = 1 L 8 9 9 9 9 9 10 E 2 2 4 L 8 8 9 w = 9 8 9 9 9 9 9 10 E 2 2 3 3 L 8 9 9 x 0 = 1 L 8 9 9 9 9 9 10 E 2 2 3 3 L 8 8 8 We assume that there exists an analogical dissimilarity AD on Σ. To extend to sequences, we augment Σ to Σ by adding a special symbol and we consequently augment the AP and the definition of AD [START_REF] Delhay | Analogical equations in sequences : Definition and resolution[END_REF]. We define an alignment between four sequences of Σ as the result of inserting some symbols in the four sentences to give them the same length on Σ . Then, the cost of an alignment between these four sequences is the sum of the analogical dissimilarities between the 4-tuples of letters given by the alignment.

We basically represent characters by Freeman code sequences. Let Σ = {1, 2, .., 16, } be the augmented Freeman code alphabet, with an AP composed of equations like: 3 : 5 :: 12 : 14 . Figure 1 presents four sequences corresponding to the pictures on the right side. These sequences are composed of elements of Σ plus capital letters representing anchorage points that will be described in the next section.

We define the Analogical Dissimilarity between four Sequences ADS(u, v, w, x) in Σ , as the cost of an alignment of minimal cost of the four sequences. We have given an algorithm to compute the ADS between four sentences and another to compute the k sequences x that give the lowest value to ADS(u, v, w, x), given u, v and w. We have shown that ADS has the same properties as the AD between objects except for the third property (section 4.1 point 3). The sequence x 0 in Figure 1 is the solution to the equation M in x ADS(u, v, w, x).

Sequence Segmentation

Aligning sequences composed only of Freeman code could lead to match strokes of different types and therefore to a non pertinent generation of the fourth stroke. In order to generate a coherent sequence we choose to use anchorage points. Practically, the set of Freeman codes and the set of anchorage points belong to the same space, where the intra-distance between the Freeman codes is lower than the intra-distance between the anchorage points, which is lower than the inter-distance between the sets. Hence, resolution by AP will first resolve analogies on anchorage points then resolve analogies on sequences between anchorage points as shown in the Figure 1. Capital letters in sequences represent different anchorage points (like E for cups, L for y-extremum), those points are represented by a bold point. The anchorage points are chosen in an appropriate way to conduct a logical modeling of letters with respect to the most stable strokes of each letter class [START_REF] Anquetil | Perceptual model of handwriting drawing application to the handwriting segmentation problem[END_REF].

Experimentations

Twelve different writers have made 40 times the 26 lowercase letters (1040 characters) inputted on a PDA. Each writer database is randomly split in four parts with 10 characters per class. We use them in a 4-fold stratified cross validation: one fourth for D10 database (260 data) and three fourth for D30 database (780 data). The experimentations are made of two phases in which simple writer-dependent systems based on Fuzzy Inference Systems [START_REF] Mouchère | Writer style adaptation in on-line handwriting recognizers by a fuzzy mechanism approach : The adapt method[END_REF] are learned.

Firstly two writer-dependent classifiers are learned for each writer on his D10 database and evaluated on his D30 database, and inversely. The two mean recognition rates are the Reference Rates RR10 and RR30, i.e. the recognition rate achievable without character synthesis. Here the RR10 is 82.3 % and RR30 is 94.5 %.

Secondly the handwriting generation strategies are tested. For a given writer, two, three, four, six, eight or ten characters per class are randomly chosen in his D10 database. Then 300 synthetic characters are generated per class to make a synthetic learning database. A writer-dependent classifier is learned with this base and tested on the D30 database of the writer. This experiment is done 3 times per cross-fold. The mean and deviation of these 3 × 4 performance rates are computed. Finally the means of these measurements are computed and correspond to the Writer Dependent Mean recognition rate (WDM) and the Writer Dependent standard Deviation (WDD).

We study four different strategies for the generation of synthetic learning databases. The strategy "Image Distortions" chooses randomly for each generation one among the three image distortions. By the same way the strategy "On-line & Image Distortions" chooses randomly one distortion among the image distortions and on-line distortions. The "Analogy and Distortions" strategy generates two thirds of the base with the previous strategy and the one third with analogical proportion. Two main conclusions can be deduced from these results. Firstly the aim of the article is achieved. Indeed with only two original characters the "Analogy and Distortions" allows better writer-dependent recognition rate than the reference RR10 and eight to be better than RR30. Secondly Table 1 shows that the three generation approaches (image distortions, on-line distortions and analogy) are complementary. Indeed the three strategies generate 300 data and we note that the richer are the distortions the better are WDM rates. Furthermore, Figure 2 shows that each strategy allows different variants of original characters not achievable by other ones.

Image distortions

On-line Distortions Analogy Original Scale Slant Speed Curvature Synthetic

Conclusion

We have shown that our two original generation strategies bring new distortions with regards to the classical image modifications. Furthermore, the combination of these three strategies keeps the writer scripts as it allows to learn efficient writer-depend recognizer with synthetic data generated from very few original characters. Future works can focus on two goals: to use the generation of handwriting with delta-lognormal synergies [START_REF] Plamondon | The generation of handwriting with delta-lognormal synergies[END_REF]; to improve the quality of synthetic data generating by analogy to avoid some very distorted characters which can impair the recognition.

  , an AP must verify: Symmetry of the "as" relation: a : b :: c : d ⇒ c : d :: a : b Exchange of the means: a : b :: c : d ⇒ a : c :: b : d Determinism: a : a :: b : x ⇒ x = b

Figure 1 .

 1 Figure 1. Resolution on Freeman direction sequences by AP and the corresponding characters.

Figure 2 .

 2 Figure 2. Examples of synthetic characters generated by the three approaches.

Table 1 .

 1 Table 1 sums up the experiment results. ± 3.3 82.5 ±2.4 85.8 ±2.0 89.4 ±1.7 91.5 ±1.6 92.7 ±1.3 On-line & Image Distortions 84.4 ±2.6 88.0 ±2.1 90.3 ±1.7 92.3 ±1.6 93.4 ±1.2 94.2 ±1.0 Analogy and Distortions 84.9 ±2.6 89.3 ±2.1 91.3 ±1.4 93.5 ±1.1 94.5 ±1.0 95.2 ±0.9 Writer Dependent Mean (WDM) performance rate (%) and Writer Dependent Deviation (WDD) (%) for different synthetic handwriting generation strategies, the reference rate RR10 is 82.3 % RR30 is 94.5 %.

	Nb. of used characters	2	3	4	6	8	10
	Image Distortions	76.1

* This work is supported by the Brittany Region and the CNRS.