A Mazyad

F Teytaud

C Fonlupt

Monte-Carlo Tree Search for the "Mr Jack" Board Game

Recently the use of the Monte-Carlo Tree Search algorithm, and in particular its most famous implementation, the Upper Confidence Tree can be seen has a key moment for artificial intelligence in games. This family of algorithms provides huge improvements in numerous games, such as Go, Havannah, Hex or Amazon. In this paper we study the use of this algorithm on the game of Mr Jack and in particular how to deal with a specific decision-making process. Mr Jack is a 2-player game, from the family of board games. We will present the difficulties of designing an artificial intelligence for this kind of games, and we show that Monte-Carlo Tree Search is robust enough to be competitive in this game with a smart approach.

Introduction

Monte-Carlo Tree Search (MCTS) [START_REF] Kocsis | Bandit based Monte-Carlo planning[END_REF][START_REF] Coulom | Efficient selectivity and backup operators in Monte-Carlo tree search[END_REF][START_REF] Chaslot | Monte-Carlo strategies for computer Go[END_REF] is a recent algorithm for decision-making, in a discrete, observable and uncertain environment, with finite horizon. It can roughly be described as a reinforcement learning algorithm. This algorithm has been particularly successful in performing abstract games [START_REF] Gelly | Combining online and offline knowledge in uct[END_REF][START_REF] Lorentz | Amazons discover Monte-Carlo[END_REF][START_REF] Cazenave | Monte-Carlo Kakuro[END_REF][START_REF] Arneson | Monte-Carlo tree search in Hex[END_REF][START_REF] Teytaud | Creating an upper-confidence-tree program for Havannah[END_REF], but has moderate results in complex games due to implementations caveats. Previous works have been done on complex games, but authors often work on a limited version of the game, generally by using only a subset of the rules. For instance, in the Settlers of Catan board game [START_REF] Szita | Monte-Carlo tree search in Settlers of Catan[END_REF] no trade interaction between players was specified, or in Dominion [START_REF] Winder | Methods for approximating value functions for the Dominion card game[END_REF] only a subset of the possible cards was used. However, more recently a competitive MCTS player has been designed for the game of 7wonders [START_REF] Robilliard | Monte-carlo tree search for the game of "7 wonders[END_REF]. In this paper, we are interested in having an artificial intelligence able to play a real game, at an amateur level, but with the full rules even when the search space becomes huge.

In this paper, we first present the game of Mr Jack in Section 2, before presenting the MCTS algorithm in Section 3. Then, in Section 4, we present the difficulties one can have for designing an artificial intelligence for the game of Mr Jack. In Section 5, we present the experiments and the results we get for the creation of an MCTS player. Conclusion and future works are then discussed. [START_REF] Bouzy | Monte-carlo go reinforcement learning experiments[END_REF] The game of "Mr Jack"

In this section, we briefly sum up the rules of the Mr Jack board game. Mr Jack is a 2-player board game which aims at recreating the Jack the Ripper story. 8 characters are used throughout the game and one is Jack.

One of the 2 players is Jack the Ripper while his opponent plays the investigator. Jack's goal is to escape the investigators taking profit of the darkness while his opponent has to catch him. Jack is the only one to know who he is among the 8 starting characters.

The game-board features the Whitechapel district and is divided into several hexagonal areas called hexes. Some of the gaslights are lit and all the hexes next to one of these are considered as illuminated, and if a character stops by a lit area he/she is considered as a seen character. This rule also applies if two characters are on two adjoining hexes. The other characters (i.e. that are not on an illuminated area or are not close to another character) are considered to be in the darkness and so to be unseen.

Each player will embody several characters. The game is divided into several turns where each player embodies one of the 8 characters. At each turn, fours characters are randomly drawn and used (two by the investigators and two by Jack). Each character can make a move from 1 to 3 hexes (see Fig. 1 for a graphical view of the board). Each character has the ability to move across the street hexes (grey hexes), the other hexes (building) cannot be crossed. When a character enters a manhole hex that is open, he can use 1 movement point to go to any other hex that has an open manhole.

At the next turn, the four remaining characters are played. Note that unlike many 2-player games, each player does not play alternatively. The first player (A) chooses a character and plays it, then player (B) chooses two characters and plays them. Finally, A plays the last remaining character. For the next turn, the play order is B A A B. Figure 1 gives an overview of the board as well as an example of seen and unseen characters. Four characters can be seen (standing next to a gas-lit) while the four others remain unseen.

At the end of every other turn, there is a "call for witness". It means that the Jack player must announce whether or not the Jack character (i.e. remember that he is the only one to know who Jack is) is visible or not. Once the call for witness has been made, the investigators' side can possibly clear some of the characters.

The game goes on for eight turns. When the detective puts a character on the same hex as another investigator/character and rightfully accuses him, the detective player wins. When a character is accused falsely, or Jack escapes Whitechapel or he manages to stay out of the hands of the police during the 8 turns, the Jack player wins.

So, the game can roughly be summed up as a deduction game. The investigator will try to narrow down his search by splitting the innocent or guilty characters into smaller and smaller sets while Jack will try to get the investigator confused by preventing him from building these sets.

However to make things much more complex each character has a special ability that modifies the rules and/allow special moves.

For instance,

-Sherlock Holmes draws a card to clear one of the eight characters.

-John Watson must shine his lantern in a certain direction and illuminate all the hexes in that direction. -Inspector Lestrade must move a police cordon that prohibits some exits on the board. -Sergeant Goodley must whistle and force some characters to move one or more hexes closer to him. -... The interested reader may refer to the full rules of Mr Jack at this location http:// hurricangames.com/datapdf/device/mr-jack-london_rules_en.pdf.

Unlike many board games, Mr Jack is quite an easy game to learn. Both players use the same tokens throughout the game but have different goals. And besides a simple aspect, Mr Jack is a deep and complex game that remains a challenge for IA applications. One of the challenges is to consider how a move of one character will affect the possible moves of many other pieces over the next turns and how to prevent the opponent to guess his future moves.

Monte-Carlo Tree Search

In this section we first present a description of the Monte-Carlo Tree Search algorithm, before detailing more specifically the Upper Confidence Bound implementation.

MCTS description

The Monte-Carlo Tree Search algorithm has been recently proposed for decision-making problems [START_REF] Kocsis | Bandit based Monte-Carlo planning[END_REF][START_REF] Coulom | Efficient selectivity and backup operators in Monte-Carlo tree search[END_REF][START_REF] Chaslot | Monte-Carlo strategies for computer Go[END_REF]. MCTS has been applied to numerous applications, and in particular in games [START_REF] Gelly | Combining online and offline knowledge in uct[END_REF][START_REF] Lorentz | Amazons discover Monte-Carlo[END_REF][START_REF] Cazenave | Monte-Carlo Kakuro[END_REF][START_REF] Arneson | Monte-Carlo tree search in Hex[END_REF][START_REF] Teytaud | Creating an upper-confidence-tree program for Havannah[END_REF].

The strength of MCTS is that this family of algorithms performs a lot of random simulations in order to evaluate a situation. Then, their use is especially relevant for applications for which it is hard to design an evaluation function. An evaluation function is a function which gives a score from a situation [START_REF] Shannon | Xxii. programming a computer for playing chess[END_REF]. Alpha-beta techniques (with the different improvements) are state-of-the-art algorithms when such a function exists. In other games, MCTS is now considered as a standard. The most known implementation of MCTS is Upper Confidence Tree (UCT). We now present this particular algorithm.

UCT description

Let us first define two functions:

s =mc(s) which plays a uniform random move m from the situation s and returns the new position s reached from s when m is played. score = result(s) which returns the final result of the situation s. s must be a final situation (no decision can be made from s). In games, such a reward is often either a loss (0) or a win (1).

The idea is to build an imbalanced partial subtree by performing many random simulations from the current state, and to bias theses simulations toward those giving good results. The construction is then done incrementally and consists of four different parts : selection, expansion, simulation and backpropagation as illustrated in Fig. 3. Sometimes in the literature, MCTS algorithm is presented in three different parts: the difference is only that the backpropagation part is presented with the expansion part.

The descent part is done by using a bandit formula, i.e. by choosing the node j among all possible nodes C s which gives the best reward according to the formula :

s ← arg max j∈C s xj + K U CT ln(n s ) n j ,
with xj the average reward for the node j, it is the ratio of the number of wins over the number of simulations, n j the number of simulations for the node j, n s is the number of simulations in s, and n s = j n j .

K U CT is called the exploration parameter and is used to tune the trade-off between exploitation and exploration. At the end of the descent part, a node, which is outside the subtree is reached. First, this new node is added to the subtree; this is the expansion part. Because this is a new node, there is no information yet, then, in order to evaluate it (as a reward is needed), a playout is performed. A playout is a random game: for both players, each decision is taken randomly until the end of the game is reached. This corresponds to the simulation part. Eventually, the reward is updated in all the nodes in the subtree which have been crossed during the selection part. This last step is called backpropagation.

The algorithm is presented in Alg.

MCTS and Mr Jack

The main challenge for applying MCTS to the Mr Jack board game lies in several aspects. First, the number of states may become really huge. Second, as explained in Section 2, one decision consists, in fact, of three steps:

-Choosing one card (corresponding to one character).

-Moving the character to a reachable location on the board.

-Applying the character's power.

Eventually, in order to get a decent artificial intelligence, game studies of MCTS have shown that the inclusion of domain knowledge can significantly improve the performance of the algorithm [START_REF] Chaslot | Monte-Carlo strategies for computer Go[END_REF][START_REF] Bouzy | Monte-carlo go reinforcement learning experiments[END_REF].

These main difficulties and the possible add of expert knowledge are now discussed.

The tree size and the branching factor

The combinatorial complexity of the Mr Jack game can be roughly estimated with the game tree size. The game tree size can subsequently be estimated by B L , where B is the average branching factor of the game, and L is the game length.

As introduced in section 2, a move in the game consists in each character making a move from one to three hexes. It is a bit difficult to evaluate how many hexes can be reached from a given position as a character may interfere with another character (by blocking a hex), a wall or a sewer can be opened or closed (thus opening or closing new short-cuts). We ran many simulations to evaluate the number of hexes that can be reached by one move. On average, a character can reach about 12 new hexes.

There are eight turns in the game and during each turn, four characters (among eight) are played. For the next turn, the four remaining characters are played. In our implementation a node is a possible move for each character, it means that for the first character in the Monte-Carlo tree, 48 nodes are created (4 possible characters × 12 possible moves), for the second among three remaining characters about 36 nodes are created (3 characters × 12 possible moves) and so forth until the second turn is reached. At this step, the last four characters are played leading to a tree of size 4 × 12 × 3 × 12 × 2 × 12 × 12 = 497664 ∼ 500000.

When two new turns are to be examined, we face a new situation where the 4 characters have not be selected yet. So all 8 characters must be considered leading to a number of possible nodes of 8 × 12 × 7 × 12 × 6 × 12 × 5 . . . × 12 =!8 × 12 8 = 1.7 13 An estimation of the game tree size for the 8 turns can be estimated by the following formula:

500000 × 1.7 13 3 = 2.5 × 10 44 (1)
Table 1(from [START_REF] Bouzy | Monte-carlo go reinforcement learning experiments[END_REF]) provides the value of B L for some well known games including the game of Go. A correlation between the size of the game tree and the difficulty to build a decent AI is known. However, equation 1 is only a very low lower bound. As explained in section 2 each character may/must use his power and this power may interfere with all characters.

Let us have a look at two different characters and see how their power can affect the game tree size. If we look closely at the Miss Stealthy character she can make a move between 1 to 4 hexes and she can cross during her move any hex including building hex. Actually, from our simulations she usually can reach almost any hex of the board. The board is actually made of 54 hexes (let us say 50 hexes to make the computations easier). It means that each time this character is used the branching factor is around 50 instead of 12.

If we look closely at the Sergeant Goodley character, the branching factor reaches new heights. This character has some kind of whistle and he must use it to call for help. Then, you get 3 movement points to use as you may slot into one or more characters in order to bring them closer to Sergeant Goodley. Moreover, this ability can be used before or after the movement. It means in the MCTS tree, each time Sergeant Goodley is used the number of nodes we have to create explode. Each character gets between 1 and 3 movement points, that means that on average some 12 hexes can be reached. Overall for this given character (12 7 ) × 2 = 71663616 new nodes are to be created (12 possible hexes, 7 characters but Sergeant Goodley and before or after the movement).

It is quite obvious that we face a difficult problem and that even the MCTS approach must be used with caution to cope with this huge search space.

Adding expert knowledge

In Mr Jack the domain knowledge is mostly driven by the powers that any character may/must use during his turn. This is quite sticky as the use of powers can change the whole set up of the game and interfere with many if not all characters. So getting a decent knowledge domain in MrJack implies an efficient use of each power.

In our implementation, we investigated several ways of including domain knowledge: an implicit approach where each power increases the branching factor of the nodes when it is used and a specific domain knowledge. We call this approach pure Monte-Carlo as the specific domain knowledge is fully embedded in the MC tree.

How to deal with the particular decisions encountered in this game ?

As we have previously seen it, in MrJack a decision consists of several parts. Precisely we have to take into account -The choice of one character among the possible cards.

-Once the character is chosen, we have to move it to a reachable position on the board. -Finally, choose to apply its corresponding power (which is different for each character).

How to deal with this kind of decisions is not an easy task. Of course, the most accurate way should be to treat all decisions separately: one decision is one character to one location with one power (the pure MCTS approach). The problem with this representation is that the branching factor becomes by far too large (see section 4.1).

Another possibility is to represent a decision as a character and one location on the board and to choose the power randomly; hoping that the impact of the position of the character on the board is more important than the impact of the power. This is a rather "crude" but "classical" way of dealing with the branching factor.

A third way of coping with the specific domain knowledge is to get some artificial intelligence for using the character ability. A simple heuristic may be used each time a character has to use his power, and a "best" move is returned. In order to keep the overall computation time to a reasonable time the heuristic for each character must be rather fast. For instance, a list of guilty and innocent characters is built as well as a list of visible and non-visible characters. Each time a character is cleared it moves from the guilty list into the innocent list. The use of the heuristic is based on both these lists. It only aims at splitting the characters of the guilty/innocent lists towards a visible or non-visible.

For example, the heuristic for Sergeant Goodley brings the characters from one of these two lists closer to a gaslight to try to make them visible or non-visible. These different representations have been experimented in Section 5.

Experiments and results

Experimental setting

We chose to experiment three versions of MCTS according to specifications introduced in the previous section. -V1 would be roughly considered as a pure MCTS scheme. The tree is fully developed and explored according to the standard MCTS rules. A node is created for every possible position of every character and possibly and the possible results of each power. Let's take an example to make the things clear. If the Sir William Gull character is to be played, (Mr Gull may exchange his location with the location of any character), and for instance, 6 moves are possible (see Fig 4). V1 will exhaustively create nodes for all 6 moves and for each result of Sir William Gull's special ability.

In fig 5 you can see that there a first node is created for move 1 and for switching Gull with character number 2, the second node is for the same move and for switching Gull with character number 3, and so on. In this case when dealing with the Sir William Gull character for these 6 possible moves, 6 × 8 nodes need to be created. (8 characters are available as it is possible for Gull not to use his ability, that means that he switches his position with himself).

-V2 would be considered as a slightly improved version of V1. When a node has to be created inside the MCTS tree and that at this node a special ability can/must be played, only one random node is generated for the ability. In a word only the movement is taken into account. It means that, for instance if the Sir William Gull character is to be played, one of the eight possible characters (including Gull) is randomly selected to switch his position with Gull.

In fig 6 you can see that only 6 nodes for Gull are created, one node for each move. The special ability is randomly chosen. It is quite obvious that such a random choice for the ability may seriously impede the AI overall level but it allows to greatly reduce the search space. In this case when dealing with the Sir William Gull character for these 6 possible moves, only 6 nodes need to be created. A random character is randomly selected to switch his position with Gull's.

-V3 is the most advanced MrJack version. Instead of randomly applying the character's ability, a set of rules is used to, hopefully, select one of the best ability of the character. These rules are a priori rules that we built and extracted. These rules have been coded using the LUA language. LUA is a lightweight programming language designed as a scripting language. LUA is quite flexible and many interpreters for the LUA language exist. We chose the LUA language as the rules for our expert knowledge program can be tuned off-line. They are easy to understand and can possibly be refined without being compiled. Furthermore it is not necessary to know the insight of the Mrjack code to modify the rules.

There is a LUA script for each character. In most scripts, the program iterates through the different characters to evaluate how far they are from the calling character. The scripts then make a decision based on these information. Based on our experiments, these scripts do not add a lot of extra computation time.

For instance, for Sergeant Goodley, the script tries to bring some characters closer or away to lit hexes so that they can possibly be cleared or not at the end of the call.

The number of nodes for the Gull character remains as low as the number of nodes needed in the V2 version.

Results and Comparisons

First, we experiment the three versions of making decisions. Results are summarized in Table 2. As we expected, the third version (V3, with knowledge for choosing the power) is the best one. The second version (V2) is roughly at the same level as the first one (V1). This shows that both power and move are an important step in making a decision.

For this experiment, we have chosen to compare versions with fixed times (1 second and 10 seconds) for each version (instead of a fixed number of simulations) because the heuristic is more time costly than the random version.

As we have seen that the third version of the MCTS algorithm is the best one, we use it for all the remaining experiments.

Wins of V3 vs V1 wins of V3 vs V2 1s 83% ± 3.76% 82% ± 3.84 10s 78% ± 4.14% 83% ± 3.76% Table 2. Percentage of wins of V3 against V1 and V2. Each experience is the average of 100 games (splited in both Jack and the inspector). As we expected the third version (V3) is by far the best version.

Tuning of the K U CT parameter

In this experiment, we have to tune the critical parameter K U CT . We have to take into account that this parameter could be different according to the player (Jack or the inspector). Each experiment is done with 10000 simulations and 200 games are performed in order to have a small standard deviation. Table 3 corresponds to the results from the point of view of the inspector, Table 4 corresponds to the results from the point of view of Mr Jack K U CT around 0.6 seems to be a good value for both Mr Jack and the inspector points of view.

Scaling on the algorithm

MCTS algorithms are known for having particular scaling properties [START_REF] Gelly | On the parallelization of Monte-Carlo planning[END_REF][START_REF] Teytaud | Creating an upper-confidence-tree program for Havannah[END_REF]. When the number of simulations increases, their impact becomes less important. The most common explanation is that there are some issues that the algorithm is not able to deal with whatever the number of simulations is. Then the algorithm reaches a plateau and increasing the number of simulations is useless. Tables 5, 6 and 7 show that this behaviour holds true for this game. We can validate in the case of the game of Mr Jack the efficiency of some well-known techniques coming from abstract games. This is another demonstration of the genericity of the Monte-Carlo Tree Search approach and we can note that some properties of the algorithm, such as the scaling perform the same way. It could really be interesting to test other known improvements such as progressive widening and Bersntein rule. The use of the famous Rapid Action Value Estimate (RAVE) improvement is not possible in this game, as there is no possibility of having a permutation of moves.

An important point should be to deal with the addition an evaluation function. We have used expert knowledge for the decision-making process, but it could be also good to use this heuristic to evaluate a situation. In the simulation step, instead of doing a random simulation, one can only play 5 or 6 random moves, and then use the evaluation function [START_REF] Lorentz | Amazons discover Monte-Carlo[END_REF]. Using determinization to simulate possible future cards in the game could also be interesting.

Fig. 1 .

 1 Fig. 1. Snapshot of the MrJack computer game. On the left, four possible characters are presented and the current player must select one of them.

Fig. 2 .

 2 Fig.2. Illustration of the MCTS algorithm from[START_REF] Browne | A survey of Monte-Carlo tree search methods[END_REF]. Circles represent nodes and the square represents the final node. On the left subtree, the descent part (guided by the bandit formula in the UCT implementation) is illustrated in bold. The evaluation part is illustrated with the dotted line. The subtree after the growth stage is illustrated on the right: the new node has been added.

Fig. 3 .

 3 Fig. 3. One iteration of the general MCTS approach from [3] .

Fig. 4 .

 4 Fig. 4. Graphical view of 6 possible moves for a given character.

Fig. 5 .

 5 Fig. 5. Graphical view of the V1 scheme for Sir William Gull.

Fig. 6 .

 6 Fig. 6. Graphical view of the V2 scheme for Sir William Gull.

Fig. 7 .

 7 Fig. 7. Graphical view of the V3 scheme for Sir William Gull.

  1.

	Algorithm 1 MCTS
	argument node s, MCTS subtree	T
	while there is some time left do
	s ← s
	Initialization: game ← ∅
	// DESCENT
	while s in T and s not terminal do
	s ← arg max j∈C s	[xj + KUCT	ln(n s ) n j

] game ← game + s S ← s // EVALUATION while s is not terminal do s ← mc(s ) r = result(s ) // GROWTH T ← T + S for each s in game do ns ← ns + 1 xs ← ( xs * (ns-1)+r) ns

Table 1 .

 1 Game Checkers Othello Chess Go B L 10 32 10 58 10 123 10 360 B L estimation

Table 3 .

 3 Tuning of KUCT . 10 000 simulations per move and V3 algorithm.

	KUCT (Inspector) KUCT (Jack) Inspector wins
	0.1	0.2	58.0% ± 3.49%
	0.1	0.3	58.0% ± 3.49%
	0.1	0.4	53.5% ± 3.53%
	0.4	0.5	58.5% ± 3.48%
	0.5	0.6	62.5% ± 3.42%
	0.6	0.7	69.5% ± 3.26%
	0.6	0.8	68.0% ± 3.30%
	0.6	0.9	72.0% ± 3.17%
	KUCT (Inspector) KUCT (Jack)	Jack wins
	0.2	0.1	57.0% ± 3.50%
	0.3	0.1	54.0% ± 3.52%
	0.4	0.1	62.5% ± 3.42%
	0.5	0.4	64.0% ± 3.39%
	0.6	0.5	65.0% ± 3.37%
	0.7	0.6	65.0% ± 3.37%
	0.8	0.6	62.0% ± 3.43%
	0.9	0.6	62.5% ± 3.42%

Table 4 .

 4 Tuning of KUCT . 10 000 simulations per move and V3 algorithm.

Table 5 .

 5 Scaling of the V3 algorithm. Point of view of the inspector. We can see that playing the inspector is easier. When the level of the algorithm is higher the game becomes more and more balanced.

	# simulations for Inspector # simulations for Jack	Wins
	50	50	87.0% ± 2.38%
	100	100	75.0% ± 3.06%
	200	200	78.0% ± 2.92%
	400	400	74.5% ± 3.08%
	800	800	69.5% ± 3.26%
	1600	1600	66.5% ± 3.34%
	3200	3200	64.5% ± 3.38%
	# simulations for Jack # simulations for Inspector	Wins
	100	50	27.5% ± 3.15%
	200	100	26.0% ± 3.11%
	400	200	28.0% ± 3.18%
	800	400	34.5% ± 3.36%
	1600	800	31.5% ± 3.29%
	3200	1600	42.5% ± 3.50%
	6400	3200	48.5% ± 3.53%
	12800	6400	40.0% ± 3.46%

Table 6 .

 6 Scaling of the V3 algorithm. Point of view of Mr Jack. 2N simulations against N. As expected, the advantage of having more simulations decreases with the number of simulations.

	# simulations for Inspector # simulations for Jack	Wins
	100	50	88.5% ± 2.26%
	200	100	81.0% ± 2.77%
	400	200	84.5% ± 2.56%
	800	400	83.5% ± 2.62%
	1600	800	71.0% ± 3.21%
	3200	1600	72.0% ± 3.18%
	6400	3200	64.5% ± 3.38%
	12800	6400	60.5% ± 3.46%

Table 7 .

 7 Scaling of the V3 algorithm. Point of view of the inspector. 2N simulations against N. Results are similar as for the point of view of Mr Jack.


Artur Aleksanyan

Etienne Brasselet
email: etienne.brasselet@ubordeaux.fr

Vortex coronagraphy from self-engineered liquid crystal spin-orbit masks

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

We report on a soft route toward optical vortex coronagraphy based on self-engineered electrically tunable vortex masks made of liquid crystal topological defects. These results suggest that a nature-assisted technological approach to the fabrication of complex phase masks could be useful in optical imaging whenever optical phase singularities are at play. The observation of faint objects near a bright source of light is a basic challenge of high contrast imaging techniques such as the quest for extrasolar planets in astronomical imaging. This has led to the development of instruments called coronagraphs, which combine the high extinction of stellar light and the high transmission of a low level signal at small angular separation. Stellar coronagraphy was initiated more than 80 years ago when Lyot studied solar corona without an eclipse by selective occul tation of sunlight, placing an opaque disk in the focal plane of a telescope [1]. On the other hand, phase mask coronagraphs offer good performance for the observation of point like sources. An early version consisted of using a disk π phase mask [2] whose chromatic drawback arising from discrete radial phase step was solved a few years later by incorporating discrete radial [3] or azimuthal [4] phase modulation to the original design. Later on, continuous azimuthal phase ramps improved the approach [5,6] and led to the advent of optical vortex coronagraphy.

Vortex coronagraphs rely on the selective peripheral redis tribution of on axis light outside an area of null intensity at the exit pupil plane of the instrument, which is done by placing a spiraling phase mask in the Fourier plane characterized by a complex transmittance of the form expilϕ, where the charge l is an even integer and ϕ is the usual azimuthal angle in the transverse plane. This enables optimal on axis rejection of light by placing an iris (called Lyot stop) at the exit pupil plane, while the off axis weak signal is almost unaffected for an angular separation larger than the diffraction limit [7]. There are two families of optical vortex phase masks that rely on the scalar (phase) and vectorial (polarization degree of freedom) proper ties of light. The former case refers to the helical shaping of the wavefront from a refractive phase mask, whereas the latter one exploits the polarization properties of space variant birefringent optical elements. Indeed, inhomogeneous anisotropic phase masks endowed with azimuthal optical axis orientation of the form ψϕ mϕ with m half integer and associated with half wave birefringent phase retardation lead to a vortex mask of charge l 2σm for an incident on axis circularly polarized beam with helicity σ 1, as originally shown in [8] using form birefringence and, in [9], using true birefringence.

The achromatic features of the vectorial option versus its scalar counterpart [10,11] have eventually led to equip state of the art large instruments such as Keck, Subaru, Hale, Large Binocular Telescope, and Very Large Telescope with vectorial vortex coronagraphs. In practice, all these installations exploit vortex masks obtained either from brute force nanofabrication, where form birefringence arises from subwavelength material structuring [8], or liquid crystal polymers technology, where true birefringence is patterned on demand [12]. Still, these high tech "writing" processes ultimately suffer from fabrication constraints and finite resolution, preventing the creation of ideal material singularity, which implies various trade offs. One can mention the preferential use of natural, rather than a form birefringence option at shorter wavelengths [10], and the use of an opaque disk to reduce the detrimental influence of central misorientation of the engineered optical axis pattern [13,14]. The realization of higher order masks with even topological charges l > 2 [5] that are desirable for future extremely large telescopes [START_REF] Delacroix | Proc. SPIE[END_REF] is another manufacturing challenge. Since the basic technological bottleneck can be identified as the man made technology itself, a nature assisted approach would likely open a novel generation of vortex masks.

Various kinds of spontaneously formed nematic liquid crys tal topological defects have been previously demonstrated to behave as natural [START_REF] Brasselet | [END_REF]17] or field induced [18 20] vectorial optical vortex generators with even charge jlj 2 and optimal beam shaping characteristics. Optical vortex masks realized with out the need for a machining technique have also been obtained from other mesophases such as cholesteric [21] and smectic [22] liquid crystals, but at the expense of efficiency. Here, we propose exploring the potential of self engineered liquid crystal vortex phase masks for high contrast imaging applications, in particular, in the case of optical vortex coronagraphy.

We choose the so called umbilical defects that appear in ho meotropic nematic liquid crystal films and negative dielectric anisotropy under a quasi static electric field and above a thresh old voltage U U of a few volts [23]. These defects are associated with optical axis orientation angle of the form ψϕ mϕ ψ 0 with m 1 and ψ 0 a constant; see Figs. 1(a) and 1(b) for imaging between crossed linear polar izers. It has been shown previously that such defects behave as spin orbit optical vortex generators [19]. This can be de scribed by neglecting the diffraction inside the optical element itself given the circularly polarized incident field in the plane of the sample, Erc σ , where r 0 refers to the defect location and c σ x iσy= 2 p refers to the circular polarization basis. In the case of transparent media as is our case, the output light field can be expressed as [24]

E out r; ϕ ∝ Er expiΔr=2fcosΔr=2c σ i sinΔr=2 expi2σψϕc -σ g; ( 1
)
where Δr is the r dependent birefringent phase retardation modeled following the work of Rapini [23]. Within such a description, one has Δr δr=r c Δ ∞ , where r c L=πK=K 3 1=2 U =U 2 1 -1=2 is the voltage dependent defect core radius with L being the nematic film thickness; K K 2 for m 1, and ψ 0 π=2 (our case) and K K 1 K 2 =2 for m 1 whatever ψ 0 , with K i1;2;3 being the Frank elastic constant of splay, twist, and bend director dis tortions [START_REF] Oswald | Nematic and Cholesteric Liquid Crystals: Concepts and Physical Properties Illustrated by Experiments[END_REF]; and Δ ∞ is the voltage dependent asymptotic value of Δ at large r. The calculated voltage independent func tion δr=r c , which describes how the retardance depends on the distance to the defect core, and reduced core radius r c =L versus reduced voltage U =U are shown in Figs. 1(c) and 1(d).

From Eq. ( 1), the spin orbit generation of a spiraling optical phase with charge 2 for the helicity flipped output field com ponent is never ideal due to the space variant retardance, which is quantitatively evaluated by the purity

η Z ∞ 0 jErj 2 sin 2 Δr=2rdr . Z ∞ 0 jErj 2 rdr;
(2) 0 < η < 1, while the spin orbit mask is described up to an un important phase factor by the complex amplitude transmittance τr; ϕ sinΔr=2 expiΔr=2 2iσψϕ:

(3)

Looking for a complex transmittance of the form expilϕ, one should ideally target an umbilic with Δ ∞ π (that defines the voltage U U π ) and a core radius r c smaller than the char acteristic beam spot size w in the plane of the sample. Here, we choose a L 10 μm thick nematic film made from a glass cell (from EHC Ltd.) whose inner surface is provided with trans parent electrodes. The liquid crystal film is prepared from the nematic MLC 2079 (from Licristal) having a dielectric relative permittivity of ϵ ∥ 4.1 along the molecular alignment and ϵ ⊥ 10.2 perpendicular to it (tabulated data at 1 kHz fre quency) and refractive indices n ∥ 1.64 and n ⊥ 1.49 at a 589 nm wavelength. The Frank elastic constant of splay, twist, and bend director distortions [START_REF] Oswald | Nematic and Cholesteric Liquid Crystals: Concepts and Physical Properties Illustrated by Experiments[END_REF] are K 1 15.9 pN (tabulated), K 2 9.5 pN (measured), and K 3 18.3 pN (tabulated), respectively. This gives U 1.83 V rms and U π 2.52 V rms [23], and r c 2.4; 2.8 μm for m 1.

In practice, once U is set above U , a few defects remain at a fixed location in the sample after the annihilation dynamics between nearby topological defects with the opposite topologi cal charges has taken place. Typically, experiments are per formed a couple of hours or more after the voltage, whose steady value is set at U π , has been switched on. Then, a defect is placed on axis in the focal plane (P F ) of a telescope, as de picted in Fig. 2(a quarter wave plates and polarizers (not shown in Fig. 2). Finally, all images are recorded by a 12 bit room temperature camera.

The coronagraphic behavior of our self engineered spin orbit masks is illustrated in Figs. 2(b) and 2(c) which show the so called "rings of fire" surrounding a dark area with a diameter 2R 1 f 2 =f 1 in the exit pupil plane P 2 . The four fold rotational symmetry for m 1 is reminiscent of the elastic anisotropy of the liquid crys tal. Indeed, while only the twist distortions are involved for m 1, when K 2 < K 1 , both chiral and splay distortions at play for m 1 [23] break the axisymmetry; see [START_REF] Clerc | [END_REF] for recent numerical investigations. Ensuing starlight rejection is demon strated in Figs. 3(a) and 3(b) which refer to the image of the on axis source that is collected in the plane P 3 when the corona graph is "on" (i.e., presence of Lyot stop I 2 ) or "off" (i.e., I 2 is removed), while the azimuth averaged radial intensity profiles in the plane P 2 are shown as solid curves in Fig. 3(c).

Quantitatively, the coronagraphic performances are gauged from the rejection of on axis illumination. A peak to peak star light rejection ratio of R peak;exp ≈ 1000 is obtained and is as sociated with a typical power rejection rate R power;exp ≈ 50 defined as the ratio between the power outside and inside the Lyot stop:

R power Z ∞ R 2 I 2 rrdr . Z R 2 0 I 2 rrdr; ( 4
)
where Simulations are carried out under the assumption of a para xial on axis plane wave impinging on the iris I 1 , namely by considering E 1 ∝ circr=R 1 as the entrance pupil field in the plane P 1 , where circr=ρ 0 for r > ρ and circr=ρ 1 for r < ρ. Then, η is calculated by inserting the expression of starlight intensity profile in the plane P F , namely I F jF E 1 j 2 , in Eq. ( 2), where F refers to the Fourier trans form. This gives η model 0.80; 0.75 for m 1. On the other hand, I 2 r jF -1 τr; ϕF E 1 j 2 , where F -1 refers to the Fourier transform, and gives R peak;model ≃ 5 × 10 4 ; 2 × 10 4 for m 1. Finally, inserting the latter expression of I 2 r in Eq. ( 4) gives R power;model ≈ 3000; 1500 for m 1. As usual, in vortex coronagraphy, simulations predict better performances than experiments. Indeed, any source of material and optical imperfections is not taken into account. Still, our experimental values are rather promising for a first at tempt, noting that early [10,27] and recent [START_REF] Absil | Proc. SPIE[END_REF] laboratory and on sky astronomical observations with an artificial vectorial vor tex mask reported a peak to peak stellar rejection ratio <100.

I
Next, we implement the coronagraphic observation of a "star/ planet" system. This is done by adding a faint off axis illumina tion (the "planet") at an angle 2α diff (α diff 0.61λ=R 1 is the diffraction limit) from the on axis light (the "star") obtained from a distinct laser source. The experimental situation without a coronagraph is shown in Fig. 5(a), where the peak to peak in tensity ratio between the star and the planet in the imaging plane P 3 is I star =I planet 20 for the purpose of demonstration. The latter ratio is drastically reduced to 0.15 when the vortex mask is electrically turned on; see Fig. 5(b). Data are compared with simulations taking the same conditions as in experiments. Namely, the images of the star and the planet are calculated from I j jF E j j 2 when the coronagraph is off (Lyot stop is removed) and I j jF circr=R 2 F -1 τr;ϕF E j j 2 when the coronagraph is on, with E star 20 p circr=R 1 and E planet circr=R 1 expi2kα diff y, k 2π=λ. More quantitatively, the coronagraphic throughput is experimentally assessed from the transmitted power ratio P on =P off of the light field passing through the Lyot stop when the coronagraph is on and off as a function of α=α diff . Figure 6(a) summarizes the data obtained for three independent defects for each m, where the markers refer to the experiment while the solid curve corresponds to simula tions, which exhibit good agreement. Moreover, the central apodization effect of the spin orbit vortex mask due to the r de pendent retardance profile [Fig. 1(a)] is illustrated in Fig. 6(b) which shows the ratio T 2α diff =T 0 between the transmission of the two identical off axis (α 2α diff ) and on axis (α 0) point sources as a function of the ratio w=r c when the corona graph is on, taking Δ ∞ π, R 1 1 mm, and R 2 0.75R 1 , whatever r c . The apodization drawback manifests typically for w < 10r c whereas, for w > 10r c , the asymptotic high contrast regime is reached. The cutoff ratio for w=r c , of course, decreases with the separation between the sources. Finally, note that present self engineered liquid crystal spin orbit vortex masks are not restricted to operating at a predetermined central wave length since Δ ∞ is electrically tunable as is the case of their ar tificial self engineered counterparts [START_REF] Slussarenko | [END_REF].

More broadly, the proposed nature assisted approach to the fabrication of singular phase masks goes beyond the possible use in optical vortex coronagraphy, and includes stimulated emis sion depletion microscopy [30] and spiral phase contrast micros copy [31]. Moreover, the proposed approach is not restricted to pure phase masks and opens the way to self engineered apod ization masks that are heavily used in coronagraphy [32], which may involve the use of either transparent or absorbing liquid crystal mesophases. Further work is needed, in particular regard ing large vortex mask size issues as usually requested in low numerical aperture telescopes.

  OCIS codes: (260.6042) Singular optics; (160.3710) Liquid crystals; (050.4865) Optical vortices.

  ), which sketches the main part of the optical arrangement used to simulate a vortex coronagraph in the lab oratory where starlight is mimicked by an on axis incident quasi plane wave. Experiments are performed using point like sources generated from He Ne lasers at a wavelength λ 633 nm and located in the focal plane of the microscope ob jectives (10×, NA 0.25) with an overfilled entrance pupil. The obtained light beams are collimated by lenses with a 20 cm focal length, apertured by an iris with 25 mm diameter, and remotely redirected at an angle α from the optical axis z by mirrors. Lenses L 1 and L 2 [see Fig. 2(a)] are identical microscope objectives (4×, NA 0.1, entrance pupil radius R 4.42 mm); the radii of I 1 and I 2 are R 1 1 and R 2 0.75 mm, respectively. This gives w 0.61λ= 0.1R 1 =R 17 μm. The imaging of the sources is made by lens L 3 with a focal length of f 3 30 cm. The input circular polarization is ensured by the polarizer and quarter wave plate, whereas the contra circularly polarized field component after the optical vortex mask OVM is selected by another set of

Fig. 1 .

 1 Fig. 1. (a) and (b) Typical images of nematic umbilical defects with a topological charge m 1 observed between crossed linear polar izers. (c) Calculated radial dependence of the function δ from Rapini's model. (d) Calculated reduced core radius r c =L versus reduced voltage U =U for m 1 (red/blue curves).

Fig. 2 .

 2 Fig. 2. (a) Optical setup. It consists of a set of irises I 1;2 and lenses L 1;2;3 with focal lengths f 1;2;3 , and an optical vortex mask of the topo logical charge l, OVM l , placed in the Fourier plane P F of the tele scope formed by lenses L 1;2 . The planes P 1;2 refer to the entrance and exit pupil planes, respectively, while the plane P 3 refers to the imaging plane. The red lines refer to the ray optics tracing. (b) and (c) Rings of fire observed in the exit pupil plane P 2 for m 1.

2 r

 2 is the intensity profile in the plane P 2 . Statistics from 10 defects for m 1 are summarized in Fig. 4. The purity η shown in Fig. 4(a) is evaluated as the ratio between the power of the generated vortex beam (post selection circular polarizer is present) and the total output power (post selection circular polarizer is removed). The experimental data are η exp ≃ 0.55 0.02; 0.46 0.02 for m 1. Statistics of measured power rejection rates are shown in Fig. 4(b) which gives R power;exp ≃ 46 9; 43 7 for m 1.

Fig. 3 .

 3 Fig. 3. (a) and (b) Typical on axis point like source images in the plane P 3 when the coronagraph is off/on, I 0 : maximal intensity in the off state, while the intensity range refers to the 4096 levels of the 12 bit camera. (c) Azimuth averaged angular intensity profile in off (black curve) and on (color curves) states. Solid curves, experimental data; dashed curves, simulations; red curves, m 1; blue curves, m 1.

Fig. 4 .

 4 Fig. 4. Spin orbit masks performance statistics. (a) Purity η of the optical vortex generation process. (b) Power rejection rate R power . The colored area refers to the standard deviation region. Red square mark ers, m 1; blue circle markers, m 1.

Fig. 5 .

 5 Fig. 5. Imaging a star/planet system when the coronagraph is off (without the Lyot stop, which gives a smaller point spread function than in the case when iris I 2 is present) and on. (a) and (b) Experimental data. (c) and (d) Simulations.

Funding.

  Agence Nationale de la Recherche (ANR) (ANR 10 IDEX 03 02).

Fig. 6 .

 6 Fig. 6. (a) Transmitted power ratio P on =P off as a function of the nor malized angular separation distance α=α diff to the telescope axis for m 1. Markers, measurements with red/blue color for m 1. Solid curves, simulations. (b) Calculated ratio T 2α diff =T 0. Insets: normalized intensity profiles of on axis (α 0) and off axis (α 2α diff ) point sources and δ in the plane P F for w=r c 1 and w=r c 100.

Acknowledgment. This study has been carried out with financial support from the French State, managed by the ANR in the frame of the Investments for the Future Program IdEx Bordeaux LAPHIA.


Yasmin Ansari

Egidio Falotico

Yoan Mollard

Baptiste Busch

Matteo Cianchetti

Cecilia Laschi

Baptiste

A Multiagent Reinforcement Learning Approach for Inverse Kinematics of High Dimensional Manipulators with Precision Positioning

published or not. The documents may come

I. INTRODUCTION

Minimally invasive surgery (MIS) [START_REF] Ponsky | Minimally Invasive Surgery[END_REF] is an advanced surgical procedure that uses a limited number of ports either natural or through a small incision to access internal organs. The use of manipulators based on traditional design (i.e. cables, pulleys, gears) for single-port intervention is known to be limited in its distal maneuverability due to few degrees-offreedom (DOFs). A more innovative approach involves the application of Soft Robotics [START_REF] Kim | Soft robotics: a bioinspired evolution in robotics[END_REF] to design whole arm flexible manipulators with virtually infinite DOF. These flexible manipulators take inspiration from boneless structures found in nature such as the octopus tentacles, elephant trunks, etc. that exhibit advanced manipulation capabilities due to their muscular arrangement. STIFF-FLOP [START_REF] Ranzani | A Soft Modular Manipulator for Minimally Invasive Surgery: Design and Characterization of a Single Module[END_REF] is an example of a soft surgical manipulator that is modular where each module comprises of radially arranged flexible fluidic actuators (FFA) [START_REF] Greef | Towards flexible medical instruments: Review of flexible fluidic actuators[END_REF] encapsulated within an elastomeric outer body. A This work is supported by People Programme (Marie Curie Actions) of the European Union's Seventh Framework Programme FP7/2007-2013/ under REA grant agreement number 608022 Y. Ansari, E. Falotico, M. Cianchetti, C. Laschi are with the Soft Robotics Lab, The BioRobotics Institute, Scuola Superiore Sant'Anna, Pisa, 56025, Italy.(email: y.ansari, e.falotico, m.cianchetti , c.laschi)@sssup.it combination of various simple feedforward actuation sequences produces elongation and omnidirectional bending. Combining three modules in series results in many DOFs with a highly dexterous workspace with no kinematic singularities. Additionally, they are safe to interact with due to the compliance of the soft actuators. However, the successful application of such soft continuum manipulators in these demanding environments is dependent upon its ability to automate low-level reaching skills with precision.

Marchese et. al. [START_REF] Marchese | Design, kinematics, and control of a soft spatial fluidic elastomer manipulator[END_REF] applied a closed-loop controller on a 3D soft-arm to position the end effector to reach a ball with a diameter of 0.04m. Giorelli et al. [START_REF] Giorelli | A two dimensional inverse kinetics model of a cable driven manipulator inspired by the octopus arm[END_REF] used a Jacobian based approach to reach an average tip accuracy of 6% the total manipulator length. These traditional methods are limited by modelling assumptions, computational expense, and most importantly, precision that needs to be further reduced for technological advancements in soft robotics. Learning mechanisms [START_REF] Nguyen-Tuong | Model learning for robot control: a survey[END_REF] provide a more promising approach by encoding correlations between sensorimotor data through internal models [START_REF] Wolpert | Internal models in the cerebellum[END_REF]. Malakzadeh et. al [START_REF] Malekzadeh | Learning by imitation with the STIFF-FLOP surgical robot: a biomimetic approach inspired by octopus movements[END_REF] applied imitation learning to a high-dimensional soft manipulator though this implies that the robot can only be as good as the provided information. Interaction with the environment through exploration is essential for a robot to learn optimal behavior [START_REF] Vannucci | Adaptive visual pursuit involving eye-head coordination and prediction of the target motion[END_REF] [START_REF] Vannucci | Integrating feedback and predictive control in a bio-inspired model of visual pursuit implemented on a humanoid robot[END_REF]. However, this is a non-trivial task for highdimensional systems that can generate a large amount of redundant data [START_REF] Magill | Motor learning and control: Concepts and applications[END_REF]. Morphological Computation [START_REF] Zambrano | The Morphological Computation Principles as a New Paradigm for Robotic Design[END_REF] has potential as a control paradigm to exploit high-dimensional structures as a computational resource through exploration, however, current applications [START_REF] Hauser | Towards a theoretical foundation for morphological computation with compliant bodies[END_REF] are limited to learn dynamical behavior without taking precision into account. Goal-directed motor exploration [START_REF] Rolf | Efficient exploratory learning of inverse kinematics on a bionic elephant trunk[END_REF] is the most optimal framework, so far, to learn inverse kinematics with precision but it requires the controller to define linear paths in statespace. We propose to address this task through Multi-Agent Reinforcement Learning (MARL) [START_REF] Yang | Multiagent reinforcement learning for multi-robot systems: A survey[END_REF] by viewing a Y. Mollard, B. Busch, are with the FLOWERS Lab, INRIA (email: yoan.mollard, baptiste.busch)@inria.fr

A Multiagent Reinforcement Learning Approach for Inverse Kinematics of High Dimensional Manipulators with Precision Positioning

Yasmin Ansari, Egidio Falotico, Member, IEEE, Yoan Mollard, Baptiste Busch, Matteo Cianchetti, Member, IEEE Cecilia Laschi, Senior Member, IEEE manipulator as a group of independent agents that share an environment where they must coordinate their behavior through autonomous exploration to reach a target. MARL has gained rapid success in multi-robot systems [START_REF] Busoniu | Multi-Agent Reinforcement Learning: A Survey[END_REF] due to its inherent nature to decentralize complex problems that accounts for a speed-up in learning. Despite these potential benefits, these applications are limited to a few DOFs [18] [19]. This is due to the 'curse-of-dimensionality' faced as the number of agents increase. An additional challenge is to find solutions in 3D continuous state-space with precision. The novelty in this work lies in the design of a reward-guiding mechanism that enables the manipulator to learn optimally increasing actions over time in abstracted state-space to reach a global optimum with precision. We test our algorithm in simulation on a 6-DOF planar robotic arm with a discrete action set. It is able to reach 129 points in its workspace with a mean accuracy of 0.0056m (±0.002). We then validate this concept on the Baxter arm in 3D Cartesian space. We show that the algorithm can reach the goals within 0.008m precision with exceptions from joint limits.

Section II discusses the development of the learning framework. Section III presents the model selection methodology followed by experimental analysis on the simulated arm and Baxter anthropomorphic platform in Section IV. Section V provides a conclusion of the overall work with future research goals.

II. INVERSE KINEMATICS LEARNING MODEL

RL [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] is an adaptive optimization technique where a single agent uses trial-and-error to learn an optimal behavior. Model-free robot control policies can be learnt through an action-value function (also called Q-function) by maximizing the expected cumulative discounted reward after executing an action (at =a) in a current state (st = s) and following a given policy 𝜋 (Є-greedy, etc.) thereafter.

𝑄 𝜋 = 𝐸 { ∑ 𝛾 𝑗 𝑟 𝑡+𝑗+1 𝜏 𝑗=0 |𝑠 𝑡 = 𝑠 , 𝑎 𝑡 = 𝑎, 𝜋} (1)
Where, 𝛾 is the discount rate and γ ∈ [0, 1]; t denotes the time-steps to when the episode terminates; r is the reward received at each time-step. This is scaled to multiple-agents by associating an independent Q-function [START_REF] Claus | The dynamics of reinforcement learning in cooperative multiagent systems[END_REF] to each agent as shown in Equation 2. A control policy can be obtained only if all concurrent policies reach a global optimum.

where, i = 1…n, where n is total number of agents. However, the presence of multiple agents within the same environment renders it non-stationary and partially-observable from the point of view of a single agent. The model-free Sarsa(ƛ) Temporal Difference (TD) with eligibility traces control approach is applied that is an online on-policy Q-function estimator with memory, allowing real-time adaptive control. Mathematically,

𝑄 𝑡+1 (𝑠, 𝑎) = 𝑄 𝑡 (𝑠, 𝑎) + 𝛼 * 𝛿 𝑡 * 𝑒 𝑡 (𝑠, 𝑎) (3)
𝛿 𝑡 = 𝑟 𝑡+1 + 𝛾𝑄 𝑡 (𝑠 𝑡+1 , 𝑎 𝑡+1 ) -𝑄 𝑡+1 (𝑠 𝑡 , 𝑎 𝑡 ) (4)
𝑒 𝑡 (𝑠, 𝑎) = { 𝛾ƛ𝑒 𝑡-1 (𝑠, 𝑎) + 1 𝑖𝑓 𝑠 = 𝑠 𝑡 , 𝑎 = 𝑎 𝑡 𝛾ƛ𝑒 𝑡-1 (𝑠, 𝑎) 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 (5)
where, 𝑄 𝑡 (𝑠 𝑡+1 , 𝑎 𝑡+1 ) -𝑄 𝑡 (𝑠 𝑡 , 𝑎 𝑡 ) is the temporaldifference error; 𝛼 is the step-size parameter (also known as learning rate in some cases); et is the eligibility trace from the backward view of the temporal difference learning where , ƛ is the accumulating trace-decay error.

A. Abstraction of 3D Cartesian State-Space

Positioning the robotic arm in 3D Cartesian state-space formulates a continuous-valued domain. We propose to solve the task of reaching by abstracting the state-space such that the euclidean millimeter distance can be locally generalized to the same actuator input. This can be achieved through function approximation where a smaller number of features are used to represent the infinite-sized region. Tile coding [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] is differentiable, stable, piecewise-constant featurebased approximator that allows to linearly approximate solutions for a non-linear system. This is achieved by partitioning the state space into multiple layers called tilings. Each element of a tiling is called a tile that allows for local generalization of state-space dependent upon the shape of the tile. These characteristics formulate the underlying motivation to employ it as our function approximator. Only one tile per tiling is activated if and only if the given state falls in the region delineated by that tile (Fig 1). The Q-function is then simply represented by a sum of the indexes of these activated tiles as, 𝑄(s, a) = ∑ 𝛩 𝑗 (𝑠)𝑤 𝑗 𝑘 𝑗=0 [START_REF] Giorelli | A two dimensional inverse kinetics model of a cable driven manipulator inspired by the octopus arm[END_REF] where j = 1…k, where k is the total number of tilings; Θ j(s) is the value (1 or 0) of the jth tile given state s; wj is the weight of that tile. This equation is applied to all the actuators i.e. for i = 1…n, where n is total number of actuators. Equation 6shows that the computational complexity of tile coding is linearly dependent on the number of tilings. The resolution of a tile is given as, 𝑅 = 𝑤 𝑇 [START_REF] Nguyen-Tuong | Model learning for robot control: a survey[END_REF] where, 𝑤 represents the width of the tile; 𝑇 represents the total number of tilings. This equation is calculated for all dimensions of the state-space i.e., for a 3D Cartesian statespace is in the x-y-z dimensions. The shape of tiles is usually problem specific as each manipulator will occupy a different reachable workspace. This information has to be provided to a programmer who will then then create appropriate shaped tiles to fit for the given workspace. However, the authors argue that the problem can be generalized by creating tiles in spherical co-ordinates (R, ⱷ, θ). The underlying principle is to parameterize the tilings such that the range in each dimension is defined as:

(i) 𝑅 = [0 max(𝑅)]; (ii) ⱷ = [0 ° 360 °]; and (iii) θ = [0 ° max(θ) °];
where max(R) refers to the length of the manipulator in a fully extended state; ⱷ refers to the azimuth which for omnidirectional bending will always be 360⁰; and θ refers to the zenith dependent upon the contraction capability of the manipulator (Note: θ represents a zenith angle measured from the vertical axis.). This defines a volumetric space within which the manipulator is free to reach any point. Knowing these three quantities, rectangular shaped tiles can be created in each dimension, hence, limiting the tiling space to the reachable workspace that can be scaled to any manipulator. Controlling the precision is then directly proportional to controlling the width of the rectangular tile in each dimension. This will be discussed in more detail in Section III A.

B. Reward-Guided Actor-Critic Architecture

We combine the episodic Sarsa(ƛ) TD with the abstracted state-space in a model-free actor-critic architecture [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] to with a discrete action-set. A generalized policy iteration (GPI) is applied where an episode starts with the arm in a resting position proceeded by each agent following the Є-greedy policy (actor) to select either optimal actions (move the arm towards the goal) or sub-optimal actions (move the arm away from the goal). The update in the Q-function (critic) by an agent are based solely on a global environmental feedback without being affected by the behavior of any other agent. We design the scalar reward dependent upon the Euclidean distance from the goal such that it has a high negative value for distances further away from the goal that progressively but discretely decreases towards target, where it receives a reward of 0. The motivation behind this reward structure is to enable the robotic arm to make goal-directed attempts [START_REF] Rolf | Goal babbling with unknown ranges: A directionsampling approach[END_REF]. The episode will end either when the goal is reached or the maximum number of trials per episode are reached.

C. Dealing with High-Dimensionality

It has been heuristically proven that decreasing the exit probabilities out of tiles with non-optimal actions improves the performance of tile coding [START_REF] Scopes | Theoretical Properties and Heuristics for Tile Coding[END_REF]. We take advantage of our reward structure to apply this concept in our algorithm. Every time it encounters an action that is rewarded with a scalar value higher than previously encountered ones, that action will be the first one taken by the system from the rest position from the next episode onwards. This process will continue until the reward before 0 is obtained. Additionally, every time a better action is selected from the rest position, the exploration rate is reduced by half until the goal is reached a certain number of times where it is made completely greedy. In the following section, we discuss the application of this algorithm in simulation and hardware.

III. MODEL SELECTION

The robotic platform under consideration is a planar 6-DOF simulated robotic arm [START_REF] Rolf | Constant curvature continuum kinematics as fast approximate model for the Bionic Handling Assistant[END_REF]. All the joints under consideration are revolute. This gives us a mapping from an Ɍ 6 motor space to an Ɍ 2 cartesian space. The total length of the robotic arm is 1m with each link equal in length.

A. State-Action Parameter Selection

The resolution (wR, wⱷ, wθ, T), step-size (α) exploration (Є), and eligibility traces (ƛ) all need to be tuned for an optimum trade-off between precision and learning time. This is a non-trivial task that in most works is done empirically [START_REF] Whiteson | Adaptive tile coding for value function approximation[END_REF] [START_REF] Sherstov | Function approximation via tile coding: Automating parameter choice[END_REF]. We also follow an empirical approach, the generalized steps of which are: For a given Є and ƛ, we first define the reachable workspace of the manipulator as described in Section II A. Then, a minimum number of tiles nR, nⱷ, and nθ are heuristically initialized in the R, ⱷ, and θ dimensions, respectively such that the resulting precision R/nR, ⱷ/nⱷ, and θ/nθ is in a centimeter range that can be improved to millimeter range through more tiling layers. Finally, a grid search is performed over nT tilings and nα stepsizes (forming an nT x nα matrix) to reach 20 random goals. The role of the discrete action set is to ensure as much reachability as possible in the defined workspace. In this work, this is found through trial-and-error.

For planar movement, we consider only x-y cartesian and corresponding R-ⱷ spherical coordinates. The reachable workspace of the manipulator is: R = [0 1m] and ⱷ = [0⁰ 360⁰]. We heuristically selected a discrete action set of 7 actions (see Table 1). Thus, there exists a total of 7e6 = 823543 possible input combinations. For Є=0.1 and ƛ=0.9, standard values found in text, we heuristically initialize 40 and 24 tiles such that the planar tiling layer of 960 tiles has a reaching precision of 0.025m and 0.261m in the two dimensions, respectively. We then perform a grid search for tilings between 2 to 32 in powers of 2 as recommended by [START_REF] Lin | CMAC-based adaptive critic self-learning control[END_REF] and 5 step-sizes of heuristically selected values, forming a 5x5 matrix. The target radius is set to 0.008m. The algorithm runs for a total of 300 episodes with 100 trials per episode which means that in one experiment, the algorithm is given a total of 3e4 trials to converge to an optimal solution for two performance criteria: (i) mean reaching error (ii) mean episodes needed for convergence. It is worth noticing that this number is 27 times less than the number of input combinations that the manipulator can select from. The optimal settings are provided in Table 1. α was found to be 0.16 and T was 4. The value of the step-size parameter implies that the arm moves roughly six-tenth of the way across a tile to the target in one update. A smaller number of steps per tile reduces the time spent on a tile in turn decreasing the risk of error propagation if the arm takes a suboptimal action in it. With these parameter settings, the algorithm was able to reach all 20 goals with a mean reaching precision of 0.0055m (±0.00186) with an average of 56 episodes required for convergence. The state space now comprises of a total 3840 tiles (4 layers * 960 tiles/layer) where each layer is offset with respect to one another.

IV. EXPERIMENTS

We experimented on 142 points in the reachable workspace. We found the algorithm to converge for 129 points with an average reaching accuracy of 0.0059m (±0.002). From these results, it is deduced that as long as we ensure millimeter accuracy in one dimension, we can meet the overall precision requirements through this algorithm. This is useful to avoid too much tiling in the ⱷ dimension which is much larger as compared to the first. The solution for all the 142 points is found to be limited due to the discrete action set, and will be taken up as a future work using continuous action spaces. Fig 3 depicts an example of the robotic arm reaching a target, its policy development, and the total accumulated reward. This policy (Fig 3 center) illustrates the goal-directed behavior mentioned in Section II B. Depending upon the distance from the goal, the manipulator initially takes actions with a lower scaler reward. As it repeatedly encounters actions with a higher scalar reward, it tends to exploit them more. Hence, over time it learns the ability to choose optimally increasing actions. As the probability of selecting better actions increases, the trials required to reach the goal decreases, ensuring convergence. This is particularly beneficial as it guides the robotic arm to perform exploration towards regions of interest without prior knowledge of the environment, but only of the system's perception of the goal in relation to its current position. This example converges at 63 episodes after testing a total of 5742 actions (Fig 3 Bottom). It is worthy that this number of actions is much less in comparison to the available 3e4 in one complete episode. The trend for the reward accumulation (Fig 3 Bottom) increases until convergence, which is as expected. We refer the readers to the complementary video for further reference.

B. Repeatability and Robustness

We repeated the tests on the workspace 5 times found it to reach the same targets with an average of 0.0061m (±0.0019).

C. Effects of taking Optimal Actions in Resting Position

In order to demonstrate why the method proposed in Section II D is essential to deal with high-dimensions, we tested 20 random goal points with and without exploiting the reward structure in the resting position. goals are reached, however, for the latter only 9 are. After, reiterating the experiment 5 times, it was found that the repeatability was guaranteed for the former approach whereas not for the latter. Fig. 4 plots the accumulated absolute value of the reward in trying to reach each goal irrespective of whether it was successful or not. Fig 4 . (Top) illustrates the rewards accumulated by the former approach and the axes has been scaled to the size of Fig. 4. (Bottom), representing the latter approach, in order to draw a relative comparison amongst both. It highlights that the maximum reward accumulated by the former approach (-5500000) is approximately equal to the minimum reward accumulated by the latter approach (-56000000). High negative reward implies that the latter method explores the state-space by taking more suboptimal actions. As a result, more error is propagated throughout the state-space which has a direct impact on the estimated Q-functions from which control policies are learned. This is why policies learned by the latter method All policy development plots observe the goaldirected behavior mentioned previously. The learning behavior in the former approach is concentrated within the first 120 episodes where the learning time increases with the increase in distance from the resting position. This is not true for the latter method where the learning time exhibits no stable identifiable trend.

V. HARDWARE

We further validated the concept presented in Section III A on the left arm of the Baxter anthropomorphic robot developed by Rethink Robotics [28]. It has 7 DOFs: two rotational joints in the shoulder, two in the elbow, and three in the wrist. This gives us a mapping from an Ɍ 6 motor space to an Ɍ 3 cartesian space. The reachable workspace of the manipulator is identified as R = [0m 1m], ⱷ = [0⁰ 360⁰], and θ = [0⁰ 46⁰]. For the model selection and experiments, the forward kinematics (provided by the manufacturer) of the robot in simulation was used in conjunction with the algorithm and a procedure similar to Section III A was followed. The optimal parameters obtained are provided in Table 3.

A total of 150 target points in 3D Cartesian state-space were created and tested. For the given discrete action-set, the algorithm was able to generate a total of 15 solutions within a state-space of approximately (0.7m x 1m x 0.80m) region. Each solution from the set was then fed sequentially to the robot for a direct comparison of the prediction to the actual outcome. This was done by obtaining the 3D cartesian coordinates from an OptiTrack Motion Capture [29] vision feedback. Within the simulator, the solutions generated had a mean accuracy of 0.006m (±0.002). Practically, it was observed that 60% of the given solutions reached the goal with a mean reaching accuracy of 0.008m (± 0.001). The rest were off the target position with a mean reaching accuracy of 0.0138m (± 0.012). This offset could be credited to calibration issues and joint-accuracies limitations.

VI. CONCLUSION

Soft Robotics applied as an underlying key technology in the design of assistive and surgical tools can provide high dexterity in the instrument. However, their practical applicability is dependent upon the development of a new framework of intelligent control strategies that can automate accurate and repeatable low-level sensorimotor skills while taking into account the non-linearity and high dimensionality of these systems. We have designed an algorithm using the MARL framework that allows open-ended autonomous exploration limited to a manipulator's reachable workspace guided by the motivation to move towards the goal as quickly as possible. We tested this algorithm on 142 points for a planar 6-DOF arm with a discrete action action-set. With a discrete action set of 7 actions, it was able to reach 129 points with a 0.0056m mean reaching accuracy. The algorithm also has been tested to be repeatable with robustness. We further validated the results on the Baxter Robotic arm. We found it capable of reaching points with within 0.008m accuracy. Future works will take into account continuous action spaces and follow trajectories. The algorithm will also be applied to soft robotic platforms.

  𝑄 𝑖 𝜋 (𝑠, 𝑎) = 𝐸 𝑖 𝜋 { ∑ 𝛾 𝑗 𝑟 (𝑗+𝑡+1) 𝜏 𝑗=0 |𝑠 𝑡 = 𝑠 , 𝑎 𝑡 = 𝑎, 𝜋}

Fig 1 .

 1 Fig 1. 2D Cartesian plane is abstracted into 2-layered rectangular tilings in R and ⱷ dimensions. There are 4 tiles with a width of wR and wⱷ in each dimension, respectively. The origin of the tilings are offset w.r.t. each other. The position of the soft manipulator end effector activates one tile per tiling.

  The base is centered at the origin of the Cartesian plane. The discrete action set can decrease/increase the angle or keep it unchanged within a range of [-180⁰, 180⁰]. The robotic arm, its initial position, and target points of the robotic arm is illustrated in Fig 2.

Fig 2 .

 2 Fig 2. The workspace of the 6-DOF robotic arm

Fig 5 (

 5 Top) require a much larger learning time in comparison to that in Fig 5 (Bottom) as well as the uncertainty of convergence in the former approach. However, it is worth mentioning that in the case the manipulator cannot deal with deal high dimensions, whenever it does find solutions, they are still high-precision. This fact is reiterated in Fig 5 which compares the policy development for the two approaches.

Fig 3 .

 3 Fig 3. (Top) An example of the 6-DoF planar converging to an optimal solution within 42.5 secs with a reaching accuracy of 3.2mm (Center) Policy Development (Bottom) Absolute Value of total reward accumulated

Fig 5 (

 5 Fig 5 (Top) superimposes the policy development plots for the 20 learnt goals in the former method, whereas, Fig 5 (Bottom) superimposes the 9 learnt goals in the latter approach.All policy development plots observe the goaldirected behavior mentioned previously. The learning behavior in the former approach is concentrated within the first 120 episodes where the learning time increases with the increase in distance from the resting position. This is not true for the latter method where the learning time exhibits no stable identifiable trend.

Fig 4 .Fig 5 .

 45 Fig 4. (Top) Rewards accumulated when taking progressively optimal actions from resting position for 20 targets. Steady-state straight lines show convergence. (Bottom) Rewards accumulated without taking progressively optimal actions from resting position for 20 target points. Linear lines indicate i.e. no learning.

  Two such examples have been provided in Fig 6 and 7 below.

Fig 6 .Fig 7 .

 67 Fig 6. Target: [R = 0.76m ⱷ = 63⁰ ⱷ = 7⁰]; (Left) Real Reaching Accuracy: 0.019m (Right) Simulated Reaching Accuracy: 0.0052m generated in 52s

Table 1

 1 A summary of experimental results on 6 DOF planar arm

  Table 2 illustrates a summary of the obtained results. For the former approach, all

	REACHABLE	R	0-1m	nR	40	wR	0.05
	WORKSPACE	ⱷ	0°-360°	nⱷ	24	wⱷ	0.26
		Є		0.1
		ƛ		0.9
		ɣ		1
	STATE-SPACE PARAMETER SELECTION	T	Range Tested Optimal Value		2 -32 (Powers of 2) 4
			Values Tested	[0.25 0.16 0.08 0.05 0.025]
		α	Optimal Value		0.16
	ACTION SET		[-0.087 -.0349 -0.0175 0 .0175 0.0349 0.087] (m)
	MEAN REACHING ERROR FOR 20 GOALS			0.0055m ± 0.00186m
	MEAN EPISODES REQUIRED FOR CONVERGENCE

Table 2

 2 Taking optimal Actions in resting position vs. not taking optimal actions in resting position. Test results on 20 random goals

Table 3

 3 A summary of experimental Results on Baxter

ACKNOWLEDGMENT This work is supported by People Programme (Marie Curie Actions) of the European Union's Seventh Framework Programme FP7/2007-2013/ under REA grant agreement number 608022.