Mehdi Mounsif
email: mehdi.mounsif@uca.fr

Sébastien Lengagne

Benoit Thuilot

Lounis Adouane

CoachGAN: Fast Adversarial Transfer Learning between differently shaped entities

Keywords: TRANSFER LEARNING, GENERATIVE ADVERSARIAL NETWORKS, CONTROL, DIFFEREN-TIABLE MODELS

In the last decade, robots have been taking an increasingly important place in our societies, and shall the current trend keep the same dynamic,their presence and activities will likely become ubiquitous. As robots will certainly be produced by various industrial actors, it is reasonable to assume that a very diverse robot population will be used by mankind for a broad panel of tasks. As such, it appears probable that robots with a distinct morphology will be required to perform the same task. As an important part of these tasks requires learning-based control and given the millions of interactions steps needed by these approaches to create a single agent, it appears highly desirable to be able to transfer skills from one agent to another despite a potentially different kinematic structure. Correspondingly, this paper introduces a new method, CoachGAN, based on an adversarial framework that allows fast transfer of capacities between a teacher and a student agent. The CoachGAN approach aims at embedding the teacher's way of solving the task within a critic network. Enhanced with the intermediate state variable (ISV) that translates a student state in its teacher equivalent, the critic is then able to guide the student policy in a supervised way in a fraction of the initial training time and without the student having any interaction with the target domain. To demonstrate the flexibility of this approach, CoachGAN is evaluated over a custom tennis task, using various ways to define the intermediate state variables.

INTRODUCTION

The development of human civilization was supported by various human features, both in terms of morphology and behavior [START_REF] Boyden | The Biology of Civilisation: Understanding Human Culture as a Force in Nature[END_REF]. Among these evolution pillars, knowledge transfer was a crucial asset as it enabled the specie to quickly embed a wide range of skills and abilities in its descendants. For intellectual tasks as well as physical activities, transfer knowledge can greatly decrease the time to reach expertise.

Reproducing this knowledge transfer within a population of robots is a highly desirable goal but is not free of challenges. Despite recent impressive Reinforcement Learning (RL) advances, current RL methods mostly focus on a single agent with a single assignment and still feature a very low sample efficiency, requiring millions of interaction steps and carefully tuned reward functions to develop suitable policies. As this process should be repeated for every distinct robot, it is straightforward to understand the relevance of transfer learning across all robotic applications.

Consequently, this work introduces CoachGAN, a new method for the fast transfer of skills between differently shaped agents. Building on the concept of expert/teacher/student where a teacher can use observations made during the expert class to guide its students although the teacher was not the initial transfer target, CoachGAN relies on a two-step adversarial process to train a student agent by using expert knowledge via a common teacher critic. Specifically, the main idea is to repurpose a GAN discriminator (the teacher) trained on a set of expert trajectories to provide an error signal for a student trying to accomplish the same task. As the very essence of this work is to focus on differently shaped entities, it is crucial that the student and the teacher can share a common understanding for the student to use the discrimina-tor evaluation to backpropagate accordingly its error. As such, this paper provides an innovative yet simple way to translate the discriminator's value into a learning signal via intermediate State Variables (ISV). The relevance of this technique is then assessed over a custom Tennis environment, using three different ways to translate the discriminator signal

RELATED WORKS

Recently, the transfer of knowledge and skills in deep learning has been a ubiquitous topic. While this domain has generated much interest from its very first moments in learning-based CV (Computer Vision) [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF][START_REF] He | Deep Residual Learning for Image Recognition[END_REF], it is now hardly expendable and has been applied to an even greater extent in NLP (Natural Language Processing). As a matter of fact, given the impressive quantity of resources required to train the most recent models [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF][START_REF] Martínez-González | Real-time convolutional networks for depth-based human pose estimation[END_REF], it is now very common to use pre-trained weights to initialize the model and fine-tune its parameters on the target task.

The omnipresence of transfer within the classic CV and NLP pools of tasks has nevertheless not particularly influenced the approaches with robotic control tasks where it is more common to rely on Reinforcement Learning (RL). RL has made impressive progress in recent years, from algorithmic advances [START_REF] Curran Associates | Proximal policy optimization algorithms[END_REF][START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF], to both simulated [START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF][START_REF] Cruz | Multi-modal integration of dynamic audiovisual patterns for an interactive reinforcement learning scenario[END_REF][START_REF] Jeong | Efficient learning of standup motion for humanoid robots with bilateral symmetry[END_REF] and real-world results [START_REF] Openai | Solving Rubik's Cube with a Robot Hand[END_REF]. In the RL paradigm, it is more common to train an agent on a given task and modify it afterward to see whether the agent's previously learned representations can be repurposed [START_REF] Trapit | Emergent complexity via multi-agent competition[END_REF][START_REF] Baker | Emergent Tool Use From Multi-Agent Autocurricula[END_REF]. Nonetheless, the fact that architectures are usually shallow (for most classical control settings, including Mujoco's physics-based environments, 3 layers are usually enough [START_REF] Curran Associates | Proximal policy optimization algorithms[END_REF]) does not encourage knowledge segmentation through a dimensional bottleneck, like in usual CV/NLP architectures. To broaden an agent's usability, some works propose entropy-based loss functions [START_REF] Eysenbach | Diversity is all you need: Learning skills without a reward function[END_REF] to increase the agent's curiosity and exploration as well as meta-Learning methods [START_REF] Zintgraf | CAML: Fast Context Adaptation via Meta-Learning[END_REF]. Although these works do enhance the relative reusability of an agent, there exist numerous real-world cases that would rather benefit from transferring knowledge from one entity to a distinct one. In [START_REF] Mounsif | Universal Notice Network: Transferable Knowledge Among Agents[END_REF], the authors propose to create a task model, independent from the agent's body, thus allowing them to transfer it from one agent to another. Although this technique is theoretically powerful, it is in practice limited by the manually-defined interface between the task model and the agents, heavily impacting its flexibility. This topic is however seldom addressed in recent works. In this view, the CoachGAN approach, through its differentiable chain, can cope with a broad scope of situations and is not limited by the task model expressiveness.

METHOD

The CoachGAN approach relies on an adversarial framework to transfer task-specific skills from one expert agent to a student agent with a different morphology.

GAN background

Generative Adversarial Networks (GANs), introduced in [START_REF] Goodfellow | Generative Adversarial Networks[END_REF], is a powerful concept that aims at estimating a generative model via an adversarial process. In this framework, two networks, the generator, and the discriminator are trained simultaneously to optimize a loss function that involves outperforming the other network. Specifically, the generator's goal is to output datapoint that is as close as possible to a target distribution while the discriminator loss depends on how much it succeeds at distinguishing the generator output from the real data distribution. Formally, the competition between the generator G and the discriminator D is the minimax objective:

min G max D E x∼P r [log(D(x))] + E x∼P g [log(1 -D( x))]
(1) where E represents the averaging operator, P r is the real data distribution, P g is the model distribution, defined by x = G(z), where z the generator input is sampled from a noise distribution (generally uniform or normal distribution).

CoachGAN

As opposed to most GANs applications where the generator is the only desired model and where the discriminator is generally discarded, the CoachGAN method uses the discriminator to permit the transfer to the student. The main concept of this approach is to define an intermediate state variable (ISV) that can be evaluated by the (teacher) discriminator, based on representations learned on the expert trajectories, and to which the student can relate. It will then be possible to backpropagate the student error within its model and optimize it based on the teacher discriminator evaluation. The ISV is critical to the Coach-GAN approach. Indeed, as the expert and the student have different morphologies, it is not straightforward how to directly compare their states/actions. The ISV role is to bridge the gap between the two agents by being the representation of a common, user-defined, intermediate state.

It would have been possible to guide the student by using the distance between the student and the expert's ISV as a loss value. However, while theoretically viable, this option is practically more complex to set up due to the discriminator pre-convergence. Specifically, in adversarial frameworks, the discriminator often trains faster than the generator, which, in extreme cases, can even prevent the generator from learning as all its samples are rejected by the discriminator. This consitutes an important drawback in most use cases and may requires extensive hyperparameter tuning. For this reason, it is more straightforward to rely on the teacher discriminator to train the student generator.

The first stage aims at training the teacher discriminator to create a relevant task critic. Using Equation 1, the objective becomes:

min G T max D T E x ISV ∼P T [log(D T (x ISV ))]+ E xISV ∼P g [log(1 -D T ( xISV ))]
(2) where G T , D T are respectively the fake expert generator and teacher discriminator, x ISV ∼ P T represents ISV vector from the expert dataset and xISV = G T (z) are synthetic ISV vectors from the fake expert generator, which sole purpose is to train the teacher discriminator. Once trained on the expert trajectories, the discriminator, when presented with an ISV, will return a scalar value, ranging from 0 to 1, indicating how likely it is that this sample was generated by the expert. Then, for the transfer, a new generator relevant to the student control dimensionality is created and paired with the teacher discriminator. In this configuration, the student loss is:

L S = ||D T (R ISV (G S (z))) -1|| 2
(3) Equation 3 states that the student generator aims at maximizing the discriminator evaluation given to a synthetic ISV-translated student generator solution.

Practically, for a given input z, the student generator solution G S (z) is passed to the differentiable operator R ISV that computes a discriminator-understandable ISV vector. Given that every function in this chain is differentiable, it is consequently possible to backpropagate the discriminator evaluation within the student's parameters for an optimization step. Figure 1 shows a schematic view of the method.

EXPERIMENTAL SETUP

Models

Training in the adversarial framework is notoriously difficult (Salimans et al., 2016b;[START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF] due to the multiple failure modes (discriminator preconvergence, generator mode collapse). Furthermore, as the teacher discriminator is expected to be used for the downstream transfer, it is necessary that the discriminator sample evaluation provides a suitable signal, learning-wise. Taking into account these various constraints, the WGAN-GP (Watterstein GAN with Gradient Penalty (Salimans et al., 2016a)) appeared as a natural architecture candidate because beyond making the generator training easier, it also formulates the discriminator evaluation in a way that enables it to be repurposed. Moreover, as a task-related vector must be passed to the networks, this work implements a conditional WGAN-GP, yielding the following formulation:

L D = E x∼P g [D(c, x)] -E x∼P r [D(c, x)] + λP g (4)
where c is the task related vector, λ is the gradient penalty weight and P g the gradient penalty:

P g = E x∼P x [(||∇ xD(c, x)|| -1) 2 ] ( 5
)
where x is a vector interpolated between x and x with an interpolation factor sampled uniformly between 0 and 1.

As the environment featured in this paper yields low-dimensionality observations, the models considered are quite thin, being composed of two hidden layers of 128 units each and ReLU non-linearities. For the generators, the random noise vector is sampled from a 16 dimensions latent space.

Intermediate State Variables

To demonstrate the usefulness of the CoachGAN approach, a custom 2D-Tennis environment was created. This environment features the playing agent, a serial manipulator equipped with a bat, and a ball thrown with an initial velocity, as shown in Figure 3a. The agent's goal is to position its effector in order to intercept the ball. As mentioned in Section 3, the CoachGAN approach relies on an intermediate state variables for allowing the teacher discriminator evaluation to guide the student. One of the main strength of this approach is that these intermediate state variables can be very diverse as long as the student can relate to them through a differentiable model. As an example, three configurations for this approach are provided:

Actual Kinematics: In this first configuration, the intermediate state variable is the effector position. Specifically, for a given ball configuration, the teacher discriminator is trained to distinguish between suitable effector position (that is, from a task expert) and the synthetic position from the fake expert generator. The student generator will output the target joint angles. To evaluate the student generator proposition, a differentiable forward kinematics (FK) model is used to translate the joint angles to the effector position that can thus be assessed by the discriminator, as shown in Figure 2. The FK model used in this configuration has no trainable parameters and is only used to keep track of the gradients.

Approximated Kinematics: This second composition still relies on the effector position, but this time the FK model is replaced by a neural network trained on the student configuration to compute the effector position given joint angles, also shown in Figure 2. Ball Rebound Speed: The effector position is a very convenient variable when considering serial manipulators. However, it may be less relevant in cases where agents exhibit a very different structure or in other types of tasks as well. Thus, to illustrate the CoachGAN method applicability, this third configuration demonstrates that it is possible to transfer task knowledge through measurements not directly accessible. Specifically, this case proposes to train the student using the ball velocity. This configuration requires an additional ball speed predictor model, trained to predict ball speed at the next step, given the ball configuration and the effector position, as displayed in Figure 3a.

Training the discriminator consequently requires an additional step, during which effector positions, both real and synthetic, are first passed through the predictor. This way, the teacher discriminator learns to classify suitable ball speeds, based on the predictor predictions, see Figure 3b.

Once the teacher discriminator is trained, it is used to train the student generator. As done precedently, the student generator proposed angles result in an effector position, computed either with an analytical or approximated model, which is then used by the predictor to compute the next step ball speed. The discriminator evaluation of this vector is finally used to optimize the student generator's weights.

Training the Expert

In the experiments presented below, a task expert is required. This expert agent is a 4-DoF serial manipulator, trained using PPO (Schulman et al., 2017), a SOTA RL algorithm. In a MDP (Markov Decision Process) version of the Tennis environment, the agent aims at maximizing the following reward function:

r tennis = α + c × (β + γ * exp(-d)) (6)
where α is a small constant that incites the agent to keep the ball over a height threshold for as long as possible. c is the contact flag ranging from 0 to 1 when a contact between the ball and the wall is detected, and goes back to 0 at the next step. β and γ are constant values used to weight the relative importance of accuracy when touching the wall. Finally, d is the vertical offset between the ball and the target on the wall. This reward function broadens the ball impact distribution, thus improving the agent versatility and ultimately providing a more diverse trajectory dataset. Figure 5 shows the mean cumulative reward evolution through training and, in the upper-left corner the impact distribution for a trained agent given various target heights.

RESULTS

Effector Position ISV

Once the expert agent is ready, it is used to generate the training dataset. For the two first configurations, that is, Actual Kinematics and Approximated Kinematics, the proposed ISV is the effector position. As such, each line of the gathered dataset features the initial ball conditions (position and speed) and the expert effector position recorded when contact is detected.

The adversarial pair (teacher discriminator and fake expert generator) is then trained on this trajectory dataset. Figure 6 shows the adversarial losses along training. These alternating curves are usual in Figure 7 displays the normalized discriminator evaluation of ISV (effector positions in this case), for a given ball configuration. As can be observed, the discriminator presents a clear preference for ISV along a line following the expected ball path. Furthermore, the discriminator places its highest confidence (strong yellow color) in an area located in the close vicinity of the dataset ground truth (the expert effector position, drawn in green), showing that it has indeed learned from the expert behavior. Once trained, the discriminator can be repurposed for training the transfer target, which is the student generator. In this configuration, the task knowledge is transferred towards a 5-DoF serial manipulator. As such, the generator output vector is interpreted as the target joint angles for the student. Then, it is necessary to translate the student generator output into the expected intermediate variables, that is the effector position, with the help of the unparametrized model. Figure 8a shows, in green, various ISV solutions given by the student generator, after training. The best solution, with respect to the discriminator, is also fully displayed. As a baseline, solutions from an untrained student are also displayed in white. As can be seen, the trained solution distribution is more compact and centered in the area most valuated by the teacher discriminator, as opposed to the untrained distribution. This demonstrates that it is possible to guide a generator with a discriminator through intermediate variables. Furthermore, it is possible to notice that the trained generator solution results in a student effector close to the original expert solution (light blue).

Approximated Kinematics ISV

Focus is now set on the Approximated Kinematics configuration. In this case, an approximation model is trained to predict the student effector position given the agent joint angles. Using the same teacher discriminator model as the Actual Kinematics stage, the student generator is trained similarly, replacing this time the agent analytical kinematic model by the trained one. Similar to the previous case, Figure 8b shows trained generator solutions and their baseline. As can be observed, the new generator solution distribution is also compact and adequately follows the discriminator preferences, demonstrating the reliability of the approximated model. While the approximated model is simple, this approach demonstrates that the CoachGAN is not limited to analytically defined models and can be used for cases where no robot kinematic model is available and must then be learned.

Ball Rebound Speed ISV

Finally, let us consider the last configuration where instead of relying on the effector position, the discriminator uses the Ball Rebound Speed to distinguish between expert trajectories and synthetic/not suitable ones. Figure 9 displays the discriminator evaluation of effector positions based on the predictor outputs as introduced in Section 4. Due to the fact that teacher discriminator training involves the predictor, the trained discriminator preferences do not overlap the distributions observed in Figure 7. However, they still strongly favor positions located closely to the expert ground truth. The student generator is then trained accordingly to the process depicted in Figure 4 5.4 Analysis this model relied on another dataset, it consequently generated a differently inclined discriminator, leading in turn to a closer player student. Still, it appears that these distributions are directly located in the ball speed direction, thus enhancing the agent chances of intercepting the ball. Furthermore, it is possible to explain the width of the distributions by the fact that only the effector position is registered on contact and that the model is not directly aware of the bat length. Nonetheless, as shows Figure 11, all generators provide educated solutions. To establish these results, 500 ball trajectories are recorded, and, for every case, each student samples one joint angles solution given the initial ball configuration. Then, a check for collision is run along the ball path allowing in fine to obtain the interception score of each model (dark blue bar in Figure 11. Now, let us consider the robots involved in the transfer experiments detailed above. While it might appear that these agents are kinematically close, it is important to note that the ISV vectors could have been generated by any robot having an end effector. Indeed, the student generator never has access to the expert geometry, thus broadening the spectrum of transfer.

Lastly, training these models requires negligible time and computation resources. Indeed, while the initial expert training requires 4 hours, the most timeconsuming configuration, featuring additional dataset and predictor needs less than 12% of total training time (26.51 minutes). Student training in itself can be completed in less than ten minutes. And, as was shown in the Actual Kinematics case, a discriminator can be reused, thus further reducing the time needed for transfer. These results clearly illustrate that the CoachGAN method does allow fast transfer of task knowledge between differently shaped entities.

CONCLUSION

In this work, a new step-by-step methodology for systematical and fast transfer of knowledge from one entity to a differently structured one is proposed. Relying on an adversarial framework, the CoachGAN technique was evaluated using various intermediate state variables (ISV) and shows that it is possible to transfer task knowledge to a student agent without interaction of this agent with the task. As explained in the experiment results, this method allows training student within minutes, while most RL tasks require several hours of training. Thus, the applications of CoachGAN are numerous, especially given the increasing availability of approximation models. While many research directions are appealing, future works will focus on the integration of agent-related constraints within the loss function to progressively bridge the gap between simulation and reality.

Figure 1 :

 1 Figure 1: The CoachGAN principle. The green frame depicts the teacher discriminator training process that is then used for training the student generator (orange frame)

Figure 2 :

 2 Figure 2: Student Generator (in orange) optimization in experiments Actual Kinematics and Approximated Kinematics. Discriminator evaluation passes through the R ISV module before reaching the generator

  (a) Training predictor to forsee next step ball speed (b) Training discriminator based on predictor outputs Figure 3: Intermediate training steps for the Rebound Speed configuration

Figure 4 :

 4 Figure 4: Student generator (in orange) training setup for experiment Ball Rebound. Discriminator evaluation goes through the predictor and R ISV before modifying generator parameters

Figure 5 :

 5 Figure 5: Cumulative reward through training and impact distribution for a set of given height targets

Figure 6 :

 6 Figure 6: Adversarial losses along training adversarial framework and represent one model improving in spite of the other. It is possible to see that the process finally stabilizes. In the upper right corner are displayed fake expert generator ISV solutions for various ball configurations, showing that this model decently approximated the real expert behavior.Figure7displays the normalized discriminator evaluation of ISV (effector positions in this case), for a given ball configuration. As can be observed, the discriminator presents a clear preference for ISV along a line following the expected ball path. Furthermore, the discriminator places its highest confidence (strong yellow color) in an area located in the close vicinity of the dataset ground truth (the expert effector position, drawn in green), showing that it has indeed learned from the expert behavior.

Figure 7 :

 7 Figure 7: Normalized discriminator evaluation of ISV (effector positions) in the task space for two given starting ball configurations. Higher values (yellow) are favored

  Sampled solutions for actual and approximated ISV in green. Baseline from untrained generator in white

  Task-space evalution by the discriminator based on predictor results

Figure 10 :

 10 Figure 10: 2D representation of the solution distribution. Successfull solutions are either overlapping expert position or in the ball path The performances of these models are displayed in Figure 10 where, for a given ball configuration, each model samples 2048 solutions. It is possible to notice that the blue and orange distribution, corresponding respectively to the analytical model and the approximated kinematics model, are very close, showing the approximated model reliability for transferring the discriminator evaluation. Moreover, these distributions overlap the expert effector position. The green distribution, representing solutions from the generator trained with the predictor pipeline is slightly broader than the two other distributions, as well as being located closer to the ball. As training for

Figure 11 :

 11 Figure 11: Interception performance and training time

ACKNOWLEDGEMENTS

This work has been sponsored by the French government research program Investissements d'Avenir through the RobotEx Equipment of Excellence (ANR-10-EQPX-44) and the IMobs3 Laboratory of Excellence (ANR-10-LABX-16-01), by the European Union through the program of Regional competitiveness and employment 2007-2013 (ERDF -Auvergne Region), by the Auvergne region and the French Institute for Advanced Mechanics.


Mehdi Mounsif

Sébastien Lengagne

Benoit Thuilot

Lounis Adouane

BAM! Base Abstracted Modeling with Universal Notice Network: Fast Skill Transfer Between Mobile Manipulators

Following recent trends, it appears that robot presence within human day-to-day lives is likely to grow and become ubiquitous. As many actors are engaged in this automation effort, it is plausible that the various cultural backgrounds of these actors will result in a broad range of different robots that will nevertheless need to perform similar tasks. Due to the excessively large number of experiences samples needed to successfully train a learning-based control policy, it would be remarkably useful to be able to efficiently transfer the skills acquired by a given agent to other, structurally distinct, robots. Accordingly, the BAM (Base-Abstracted Modeling) methodology proposed in this paper is a fast transfer learning approach that relies on a clear segmentation between the task model, that is a learned policy for solving a specific task and the learned robot control policy. The evaluation on two manipulation tasks using twelve different configurations of mobile manipulators demonstrates the strong potential of this approach as the segmentation results for more robust policies than naive methods and that an efficient transfer can be done in a fraction of the initial training time.

I. INTRODUCTION

The rise of human societies was supported by various specific human traits, such as the human brain size or the opposable thumb. A key feature of this expansion was however our ability to share and transfer knowledge to our peers. Indeed, seen from a global point of view, shared insights allowed important population clusters to coordinate their actions and achieve goals that are beyond the reach of a single individual, while, on a more local scale, it enables unexperimented individuals to quickly learn from domain experts, thus fastening the generation process of capable elements of the society.

Reproducing this process of knowledge transfer within a population of robots is a highly desirable goal, due to resources needed to train a learning-based agent. For instance, in the context of an industrial chain, as shown in Figure 1, where several robots are liable to perform a similar assignment, it would be interesting not to have to train each robot separately, but instead, create a task knowledge module that can be shared between different instances, even if their kinematic structure (i.e: number of DoF, segment lengths) is different, thus reducing the time for recovering an operational process, should one robot be damaged for instance.

The BAM (Base-Abstracted Modeling) method proposed in this work focuses precisely on this transfer issue as it introduces a methodology to transfer knowledge between 1 Université Clermont Auvergne, CNRS, SIGMA Clermont, Institut Pascal, F-63000 Clermont-Ferrand, France. mehdi.mounsif@uca.fr 2 Université de Technologie de Compiègne, CNRS, Heudiasyc, F-60200 Compiègne, France two differently structured entities. Specifically, it relies on a staged training process that creates a robot-agnostic, independent task model called the UNN (Universal Notice Network) that can then be seamlessly paired with a robotspecific controller (called the base modules). This enables the latter robot to solve a task by using the knowledge created by another agent, with minimal fine-tuning. In this setting, the BAM method is an environment modeling strategy that was designed to tackle potential issues that may arise when the UNN learns how to solve the task while using a poorly-conditioned robot module.

Fig. 1: Industrial integration of the Universal Notice Network approach

The UNN/BAM method capabilities are demonstrated in the context of mobile manipulators. Using three mobile bases (omni-directional, bicycle and differential), four robotic arm configurations and two environments (Pick'n Place task and the Stacking task), the experiments, set within a RL (Reinforcement Learning) framework, detailed below show that the concept of task knowledge advantages are twofold. Firstly, it is beneficial to the initial learning process, as it eases the reward function design and prevents the RL agent to fall within local minima. Next, it is particularly suitable for transferring skills between differently shaped entities, outperforming naive transfer both in final performances and training time perspective and even enabling zero-shot transfer in a particular case.

II. RELATED WORKS

Perform complex motions to complete a task is still an open issue. While there exists optimization methods based on complete models [START_REF] Lengagne | Generation of whole-body optimal dynamic multi-contact motions[END_REF], these are usually computationally intensive and may not satisfy real-time constraints. To tackle this issue, the community has also been leveraging predictive methods that are completely suitable for real-time control. However, this execution efficiency is usually achieved by sacrificing the model complexity and thus considering simplified versions that may not have enough degrees of freedom to express the full problem spectrum [START_REF] Diehl | Fast direct multiple shooting algorithms for optimal robot control[END_REF]. These methods present the strong and undeniable advantage of allowing direct transfer of skills between different agents. However, while it is possible to conceive hybrid controllers that can mitigate each approach's weaknesses, they nevertheless rely on an analytical model for both the task and the robot that can be hard if not impossible to define. As a result, it is common to integrate various hypotheses and assumptions that highly impact the precision of the model.

Data-driven approaches, in particular RL (Reinforcement Learning) have intensely been discussed in recent works as these methods are model-free [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] and are able to generate control policies for highly complex and non-linear tasks [START_REF] Openai | Learning dexterous in-hand manipulation[END_REF], [START_REF] Zeng | Tossingbot: Learning to throw arbitrary objects with residual physics[END_REF]. Nevertheleess, training these performant policies require such an important computational training budget that the benefit of task knowledge transfer become obvious. However, as opposed to usual practices in CV (Computer Vision) and NLP (Natural Language Processing) [START_REF] Tang | Distilling task-specific knowledge from BERT into simple neural networks[END_REF], [START_REF] Redmon | Yolov3: An incremental improvement[END_REF], RL architecture (being shallow and without dimensional bottleneck) are not designed for knowledge transfer to other entities. This usually results in an entangled knowledge representation. As a matter of fact, the RL paradigm focuses rather on training a single couple agent/task. In some cases, the author investigate whether the agent's previously learned representations can be repurposed by modifying slightly the task [START_REF] Trapit | Emergent complexity via multi-agent competition[END_REF], [START_REF] Baker | Emergent Tool Use From Multi-Agent Autocurricula[END_REF]. In this view, to broaden an agent's usability, some works propose entropy-based loss functions [START_REF] Eysenbach | Diversity is all you need: Learning skills without a reward function[END_REF] to increase the agent's curiosity and exploration as well as meta-Learning methods [START_REF] Zintgraf | CAML: Fast Context Adaptation via Meta-Learning[END_REF]. Although these works do enhance the relative reusability of an agent, they are still directly aiming at a single agent/single task frame while there exist numerous real-world cases that would rather benefit from transferring knowledge from one entity to a distinct one. This issue lacks representation in literature and to the best of our knowledge, there is so far no work that focuses on transferring task knowledge between differently shaped entities.

III. UNIVERSAL NOTICE NETWORK

The UNN (Universal Notice Network) presented in [START_REF] Mounsif | Universal notice network: Transferable knowledge among agents[END_REF] is a framework that implements the idea of knowledge segmentation between the agent and the task. Using 2D robots, the authors of this work show that it is possible to pair a policy learned using RL approaches with analytical controllers to achieve and transfer simple tasks. The main UNN principle is derived from the idea that a task can be achievable by any agent, by following the right set of instruction, provided the agent is able to move adequately. For this last constraint to be satisfied, the agent is pre-trained on a primitive task that enables it to acquire basic motor skills. This process is shown in Figure 2. Once trained, the UNN module can be transferred to a structurally different agent from the one used to create it and consequently enable this new agent to perform the task without prior interaction with it. In Figure 3, stage I corresponds to the initial choice of task and agent. The computation pipeline is created in stage II by pairing the trained task UNN and the base module of the performing agent (this agent is different from the one that initially constructed the UNN). This makes possible the achievement for this task by the new agent as shown in stage III. The next paragraph describes the computational pipeline set up, while the rest of this section provides details on the modules creation. u , the UNN (blue), is designed to be robot-agnostic and specific to the task T . Specifically, the state vector that is observed at each timestep, can be split into two parts s i,r , s T , respectively holding data intrinsic to the considered robot and task-related information, independent from the agent. The agent specific state s i,r is used by its input module to compute a state agent representation:

s i ,r = m r i (s i,r ) (1)
The UNN (i.e.: the task module), conditioned by the task observation vector s T , uses the processed agent representation vector to compute a vector that can be seen as a high-level instruction:

o out = m T u (s T , s i ,r ) (2)
Finally, the effective robot action is then recovered by concatenating the initial intrinsic vector s i,r with o out and feeding it to the output base, that is:

a r = m r o (o out , s i,r ) (3)
Fig. 4: The UNN Pipeline is composed of three stages: the input and output bases (in green and red) are specific to the robot, while the UNN (blue) is task-related

In the cases discussed below, the UNN and the ouput module are neural networks, but in practice, there is no theoretical restriction on the computational model used, as long as the vector returned by the modules has the expected dimensionality. From a functional point of view however, to ensure that the UNN is compatible with transfer, it is necessary to design the vector s i,r that is passed between the UNN and the output module in a way that ensures the Markov property is satisfied: it must encompass enough information for selecting the next action without needing to consider additional previous steps. The same logic is applied to the vector passed between the input module and the UNN.

B. Base Abstracted Modeling

Ultimately, the goal is, for any UNN/agent couple, to find the module functions m r i , m T , m r o , that produces the movement M generated by actions a r provided by the pipeline for solving a task τ and that must ensure the set of constraints g τ to perform the task and the set of constraints g r to ensure the physical limits of the robot, such as:

find m r i , m T , m r o such as g τ (M) ≤ 0 g r (M) ≤ 0 With: M = f (a r ) = f (m r o (m T u (s T , m r i (s i,r )), s i,r )) (4
) However, due to the RL tendency to fall into local minima, the UNN can discover a successful strategy for solving the task that depends on its body configuration (for instance, blocking an object between two articulations). In these cases, Equation 4 is longer respected. While this is not an issue for the current agent, it is detrimental for the efficiency of a future transfer to an agent with a different structure. To ensure that the UNN constraints are not entangled with the agent's, the Base Abstracted Modeling (BAM) is introduced. This approach relies on a environment modification that assimilates the robot to its effectors, thus creating a virtual robot, as shown in Figure 5. On the formulation side, we set the bases models m r i , m r o as identity functions and provide directly the UNN output to the command. These two modifications release most of the constraints that would have been distilled by the pre-trained base and prevent the UNN from learning a configuration-specific strategy, resulting in the creation of a UNN closer to a model-agnostic setting. Fig. 5: The BAM version of the environment allows the UNN to be bias-free

C. Learning techniques

In the experiments presented below, the UNN is optimized via a combinaison of reinforcement learning methods. The main technique used is PPO (Proximal Policy Optimization [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]), a policy gradient algorithm that maximizes the expected sum of rewards:

J(θ) = E[ inf t=0 γ t × r t ] (5)
where r t is the reward obtained at time t by the stochastic policy π θ parameterized by a vector of parameters θ, γ is a discount factor and E represents the averaging operator. PPO also relies on an actor-critic with trust region to improve the stability and robustness of the gradients estimation. While RL excels in reactive environments, it is usually less straightforward to design reward functions for sequential environments like those presented in Section IV-B. Indeed, in these cases, tuning the reward function in order to balance between a negative reward that may induce divergence in the agent's behaviour and a positive reward that may lead the agent to accumulate rewards without a clear line of action is very subtle and requires a considerable amount of tweaking. Thus, in order to improve the learning performances, PPO is coupled with two additional techniques:

• Pre-training with a Behavior Cloning based [START_REF] Goecks | Integrating behavior cloning and reinforcement learning for improved performance in sparse reward environments[END_REF] approach • Inverse Reinforcement Learning [START_REF] Ho | Generative adversarial imitation learning[END_REF] These two techniques, that seamlessly integrate with UNN/BAM approach, leverage a small dataset of expert demonstrations (in our case, a human operator) and provide a supplementary signal for guiding the agent within the environments.

IV. EXPERIMENTAL SETUP

A. Robots

We consider three mobile bases configurations (Omnidirectional, bicycle and differential), as well as four different manipulator types (KUKA, BLUE, Generic 2, Generic 3). Which, in total, leaves us with 12 possible configurations, three of them are displayed in Figure 6. The bicycle mobile base is non-holonomic and relies on two values: thrust and steering, for moving. The differential base is also nonholonomic and uses two torque values to compute the resulting direction. Finally, the omnidirectional base is the simplest as it is holonomic and the two values determine linearly its next position. Concerning the manipulators, both the KUKA and BLUE robots have 5 DoF. The Generic arms (rightmost configuration in Figure 6) have 2 DoF per articulation, which leads to a total of 4 and 6 DoF for the agents considered through our experiments. All serial manipulators are controlled using joints target position.

B. Environments

This section presents the two environments, implemented in the ML-Agents framework (introduced in [START_REF] Juliani | Unity: A general platform for intelligent agents[END_REF]), that were created in order to demonstrate the UNN and the BAM advantages. An additional environment, the Reaching environment is used to train the base modules and is presented below. For each of these environments, we explain the general goal, define the task-related states and detail the extrinsic reward signal used to guide the policy in the environment.

In these environments, the UNN expects a vector s i from its input base and always outputs the same type of information o out . In practice, we have:

• The processed intrinsic information s i ∈ R 6 consist of the end effector linear position and velocity, computed by an analytical approach. Specifically, the input base relies on a forward kinematic approach to compute the processed intrinsic information. • The UNN outputs o out ∈ R 3 × B, composed of desired effector linear velocity and a Boolean for sucker action. Reaching Task: This is the primitive base module training environment. Consequently, the state and actions of this specific task are agent-related. This task involves touching a given point in space with the agent's effector. The target point is regularly moved in order to force the agent to use its mobile base to get in range. This task features the following MDP:

• State: s t ∈ R 16+n : mobile base pose ∈ R 3 , arm pose ∈ R n (n joints positions), vector to target ∈ R 3 , LIDAR sensors ∈ R 10

• Action: a t ∈ R 2+n : mobile base movement vector ∈ R 2 (specific to each mobile base configuration) and target joints speed ∈ R n . • Reward: r t = -α × d E,T + i β i × c i where d E,T is the distance between the effector and the target, α is a normalizing constant and β i ×c i is a weighted constraint on the arm to prevent it from falling into undesired local minima For training the UNN and evaluating the BAM approach, we created two custom environments. Pick'n Place Task: the agent is expected to pick up an object from one table and drop it to a given point in space. In this case, the task-related state is s pap ∈ R 6 × B, composed of the current object position, the drop target position and a boolean for signaling whether the cube is held or not. The agent relies on the following reward function:

r p = α(d 0 + d 1 ) -1 ( 6
)
where r p , the reward for the Pick'n Place environment, depends on α a positive scaling factor, d 0 the distance from the object to the effector and d 1 the distance from the object to the drop target position.

Stacking Task: the goal is to stack a number of cubes. The task-related state is conceptually similar to the Pick'n Place, that is s stacking ∈ R 3 * c × B 4 where c is the number of cubes to stack, in our case c = 4. We also design a reward that penalizes the agent according to the summed distance between each of the cube and their ideal position. That is:

r s = [ 4 i=1 α i (d 0,i + d 1,i )] -1 (7)
where r s , the Stacking task reward, depends on α i a positive scaling factor that is adjusted according to the agent progression in the task, d 0,i the distance from the cube i to the effector and d 1,i the distance from the cube i to its target position.

V. RESULTS

In this section, the results of our method on both training and testing setup, over all environments are presented. In particular, we compare our architecture, the BAM modeling, with several baselines, namely the naive PPO approach and the UNN segmentation. A video of our results is available at https://bit.ly/324Sxnz.

A. Training

During training, we monitor the agent's evolution through the mean cumulative reward per episode. This metrics is common in RL and serves as a measure of how well the agent behaves within its environment.

Curves in Figure 7a are specific to our architecture and measure the output base training performances with respect to the reaching task. Figure 7b and Figure 7c respectively display these results for the whole architecture (bases + the UNN is able to learn the task more efficiently than a naive agent. As a matter of fact, we can see that both UNN-based techniques, in red and orange, respectively representing the UNN and the BAM approaches, clearly outperform the naive PPO approach, in blue. We note that the BAM (red) and UNN (orange) curves both reach a higher final cumulative reward than the naive PPO, while also being faster, indicating a better performance at solving the task overall.

B. Transfer

UNN-based approaches aim at making the skills transfer between agents of diverse constitutions possible. Hence, after training, we pass pre-trained models on a given configuration to another one with a different structure and compare their performances. In an ideal case, pairing the UNN would not require any fine-tuning. However, in practice, most cases require additional training iterations for the UNN to complete the adoption by another agent configuration. We display some examples of this process in Figure 8a and Figure 8b, respectively for Pick'n Place and Stacking environments. These curves show for each environment how fast the performance can be recovered for a given configuration. Specifically, the y-axis measures the relative mean reward of the current agent projected in the lowest-to-highest interval from all initial agents. The x-axis also indicates how much of the initial training time was necessary to reach said performance. For the Pick'n Place environment, see Figure 8a, it appears that transfer within the naive framework is not efficient and even detrimental to the agent. The policy weights when transferred to another agent do not provide a decent initialization point and, through our experiments, these agents systematically failed at the task. In the contrary, using the UNN-based techniques allows to jump-start the new agent configuration with an initial level of performance that exceeds 70% of the previous agent and allows to recover almost the full performance, evaluated by the mean cumulative reward per episode, in a fraction of the previous training time, mostly depending on the base conditioning. For the Stacking environment, similar observations emerge from the Figure 8b where, again, the UNN-based methodology allows agents with different body configuration to access to the task knowledge, thus enabling quicker performances recovering.

While mean reward is a very common metric in RL, we also consider test time performances by averaging over 100 runs the number of steps necessary to complete the task for various agents in both environments. In this setting, the best agents are the fastest and need the lowest number of steps to complete the task. Table I details results of this evaluation for Pick'n Place environment. In this case, we observe very contrasted results. No naive PPO agent was able to succeed in the task after being transferred, while the UNN-based transfer was particularly beneficial. Indeed, in these configurations, the performance was mostly conserved, with differences being related to the output base quality and movement capabilities (that is, it takes more time for the bicycle mobile base to reach a specific point than it does for the omni-directional base.) This is further confirmed by results presented in Table II, concerning the Stacking task, where test performances are globally similar for all UNN-based agents with a transferred task-model. While these results are not sufficient to unequivocally determine an optimal configuration, they do however show that in both environments, the UNN and the BAM method yield an increase in performance and allow transfer between entities. This is due to the fact that, in this modelling approach, the UNN can focus on the task at hand, without having to learn simultaneously how to correlate external perceptions to lowlevel orders.

C. Discussion

Through our experiments, it clearly appears that the UNN output base's quality is paramount for establishing acceptable performances. Indeed, the overall UNN performances, both for learning and transfer are tightly tied to the stability and robustness of the bases. More specifically, RL-trained bases do not generalize very well to a distribution of states that does not match the initial training distribution. As a result, it is difficult to predict the agent's behavior for remotely visited states. To prevent this and increase the base reliability, it is important to ensure that the base training visits the statespace regions that will be required to perform the task. This additional precaution implies that the primitive task should be carefully crafted and leads to wonder whether this whole setup is necessary. The visual observation of a naive agent provides some clues to address this concern. As a matter of fact, the naive version relies solely on the reward to guide the agent's conduct, and, even though these agents are likely to find a way to perform the task, no control or constraint are included, which might result in unwanted body configurations. While it is theoretically possible to include behavioral terms in the reward function, it is difficult to foresee the effects of such constraints and, in practice, this accumulation of weighted reward terms provides a noisy signal to the agent, drowning the task reward signal with the penalties for unwanted or encouraged behavior, hence usually minoring the performance. Using the UNN architecture provides a clear improvement from this point of view because the base's training can be more thoroughly controlled, thanks to the carefully crafted primitive task, without being distracted by the task reward, thus resulting in a more reliable agent.

This analysis was the starting point for the BAM introduction that lessens the difficulty for learning the tasks for the UNN, placing instead the burden on the interface UNN-base. Through this segmentation and a meticulously designed primitive task, it is possible to create more efficient policies that outperform the naive PPO agent on a variety of environments.

VI. CONCLUSION

The BAM approach presented in this work is a fast transfer learning method that dissociates the agent's control logic from the task it is supposed to accomplish. This technique also introduced an environment modeling modification that tackles issues related to RL-policy learning bias, making the UNN method more scalable and robust. We assessed the viability of this technique, using two custom 3D dynamic environments and several agent configurations, by comparing it with a current state-of-the-art algorithm and demonstrating that the task model segmentation implies strong benefits both during the learning phase, as the performing agent is able to learn faster and reach higher rewards by not being distracted by low-level parameters, and also during transfer. Specifically, the transfer, which is likely to fail using naive methods is not only possible using the BAM method but quite fast, even enabling zero-shot transfer in some cases. Future works will investigate the creation of a structural distance metric to evaluate how different two robots can be to still benefit from the transfer and whether it is possible to project the agent state representation within a shared latent space to avoid manually crafted representations.

Fig. 2 :

 2 Fig. 2: The original UNN Pipeline staged training first creates a primitive agent controller (in blue) for a chosen agent configuration. The UNN for the task relies on the generated primitive motor skills to learn a successful policy (in red).

Fig. 3 :

 3 Fig. 3: The successive steps for deploying the transferred UNN to a new configuration

Fig. 6 :

 6 Fig. 6: A subset of the possible agent configurations. From left to right: Bicycle KUKA, Omni-directional BLUE, Differential G3. Parts can be exchanged between agents.

Fig. 7 :

 7 Fig. 7: Learning performances over various environments

Fig. 8 :

 8 Fig.8: Fine-tuning performances using various architectures

TABLE I :

 I Transfer: Pick'n Place

	Archi.	Config.	Agent	Training %	Perf.
		Initial	Omni. Generic 2	100%	1355.4
	PPO		Omni. Generic 3	40%	Fail
		Transferred	Bicy. KUKA	40%	Fail
			Diff. BLUE	40%	Fail
		Initial	Omni. Generic 2	100%	848.8
	UNN		Omni. Generic 3	20%	784.8
		Transferred	Bicy. KUKA	20%	801.7
			Diff. BLUE	20%	443.6
		Initial	-	100%	412.3
	BAM		Omni. Generic 2	20%	751.6
		Transferred	Bicy. KUKA	20%	707.9
			Diff. BLUE	20%	747.7

TABLE II

 II

			: Transfer: Stacking
	Archi.	Config.	Agent	Training %	Perf.
		Initial	Omni. Generic 2	100%	4571.9
	PPO	Transferred	Omni. Generic 3 Bicy. KUKA	40% 40%	Fail Fail
		Initial	Omni. Generic 2	100%	3547.6
	UNN		Omni. Generic 3	20%	4004.1
		Transferred	Bicy. KUKA	20%	3966.1
			Diff. BLUE	20%	3847.9
		Initial	-	100%	2879.3
	BAM		Omni. Generic 2	20%	3540.4
		Transferred	Bicy. KUKA	20%	3391.8
			Diff. BLUE	20%	3312.7

ACKNOWLEDGMENT

This work has been sponsored by the French government research program Investissements d'Avenir through the RobotEx Equipment of Excellence (ANR-10-EQPX-44) and the IMobs3 Laboratory of Excellence (ANR-10-LABX-16-01), by the European Union through the program of Regional competitiveness and employment 2007-2013 (ERDF -Auvergne Region), by the Auvergne region and the French Institute for Advanced Mechanics.


Pascal Daloz

Patrick Johnson

Sébastien Massart

Dassault Systèmes

Pascal Le Masson

Benoît Weil

SHAPING THE UNKNOWN WITH VIRTUAL UNIVERSES-THE NEW FUEL FOR INNOVATION

The new logic of financing innovation: from uncertainty reduction to shaping the unknown Handbooks in finance, as well as literature reviews, recall that financing innovation and financing productivity investment differ in their level of uncertainty. 1 Students learn that financing production investment requires a positive net present value (NPV), whereas financing innovation requires taking into account multiple uncertainties by computing expected NPV. Models of decision-making in uncertainty helped to compute the value of reducing uncertainty. 2 This approach is considered the best way to value investment in research and development (R&D)-R&D being considered an activity to reduce uncertainty. 3 In this time of "disruptive innovation" in the context of multiple socioeconomic and technological changes-such as energy transition, aging, and digitalization-it is tempting to consider that innovation dynamics tend to be characterized by an increase in uncertainty. Investments would, therefore, become much riskier, and financing might seem almost impossible. Fortunately, this "wisdom" misses a critical feature of contemporary innovation: it is not mainly about uncertainty but much more about "the unknown". In contemporary innovation, one has to deal not only with uncertain events, such as unstable markets and technological advances, but also partially unknown chimeras, such as inclusive mobility, smart cities, and sustainable energy. Therefore, it is critical for innovation success to deal with these initially unknown situations and shape them in a beneficial direction. 4 This distinction between uncertain and unknown has major consequences on innovation investment: the financing approach must not only consist of reducing uncertainty but also of shaping the unknown, i.e., through a capacity to design new alternatives, worlds, opportunities, markets, and usages.

Paradoxically, shaping the unknown is not necessarily "worse" in terms of risks and financing. While an increase in uncertainty might lead investors to become gamblers, dealing with the unknown requires investors to understand design logic and adopt a perspective on the new potentialities to be explored. If markets and technologies are unknown, good design does not consist of multiplying risky trials-it consists of designing technologies and markets that correspond to a winning lottery. One critical result of recent advances in design theory is that the unknown, forward-looking statements might become self-fulfilling and performative; they create a common language that supports innovation. Confronted with sacrificial dilemmas, where all given decisions seem doomed to unacceptable uncertainties, design logic enables the design of new and better decisions in the unknown. 5

Risk from uncertainty versus risk from the unknown

Let's give a simple illustration of the difference between risk from uncertainty vs risk from the unknown: Famous French cartoonist Jacques Rouxel imagined strange creatures, called the Shadoks, whose rockets had one chance in a million to succeed. Consequently, they "rushed to fail the first 999,999 first trials". By contrast, design logic consists of shaping the unknown to redesign a rocket that has a 100% chance to succeed-which is actually what is expected from engineering designers in disruptive innovation! Investing in deep tech today would require similar reasoning. Deep tech refers to research-based technologies whose market applications are largely unknown, with each market opportunity having a very low probability of success. But what if designers were able to design a so-called generic technology that, for example, might be generic to several markets? Then the probability that at least one market succeeds becomes high, since the probability that all the markets fail is low. This shows how design logic differs from a gambler's logic. 6 It also explains the success of platforms that are ecosystems based on a generic technological "core" made available to multiple complementors for multiple micro-applications.

Virtual universes to explore the unknown

To support this transition from uncertainty reduction to shaping the unknown, new financing approaches and new investment models are required. How do we act in the unknown? Can we orchestrate large teams developing breakthrough innovation despite the lack of knowledge on the necessary steps? How can we just pretend that all of these people are working on the same project? And why would an investor finance such a project? Virtual universes are the keystone for these new collective behaviors. They do more than provide a shared representation: as a tool for shaping the unknown, they provide a shared capacity to present the unknown. In addition, they provide an objective basis for a comprehensive discussion of every aspect; even though the considered objects are not physically there and may be inconsistent from a scientific perspective, virtual representations don't have to obey every law of nature to be useful. Virtual universes act, therefore, both as a factual proof point-the dimensions of the virtual object can be objectively "measured"-and as a political or managerial reference, because their power comes from people believing in their performative value. In this perspective, the virtual world is not a computer game. Virtual exists because it extends and improves the real world.

Advances in research on generativity logics and design theory have shown two critical results, hinting at how virtual universes are key resources to enable and catalyze the exploration of the unknown: 1) Exploration is doomed to severe fixations-both individual and collective onesprovoking orphan innovation phenomena and speculative bubbles. But once fixation is overcome, then risk is considerably lowered by the fact that rigorous exploration of the unknown leads to the discovery and generation of diverse opportunities-across short-and long-term horizons with low and high capital expenditures (CapEx). Hence, exploration capabilities and methods that help overcome individual and collective fixations are a key resource. Today, some business units have developed such capabilities of "unknown shaping". Preliminary statistical studies analyzing their profitability show surprising results: a recent case study showed how one invested euro can bring 6 euros back to the corporation, and more than two-thirds of projects initially considered "too uncertain" to be funded are made profitable by rigorous design methods. 7 Because they provide an objective anchor for vivid imagination, virtual universes are crucial for overcoming fixations and, therefore, accelerate explorations.

2) In "unknown exploration", a critical resource is independent knowledge. This is counterintuitive for two reasons. First, it means that the unknown cannot be shaped on a "blank slate"-it requires knowledge and expertise. Second, knowledge is much more valuable if it is not self-evidently related to the issues to be explored. This second aspect is counterintuitive because, in a model of uncertainty reduction, the value of knowledge comes from dependent variables-if Y depends on X, then knowledge of X enables us to reduce the uncertainty of Y. By contrast, in design, the value of knowledge comes from independent variables-if the known Y is independent of X, then knowledge of X enables the design of disruptive Y. 8 The confrontation of these diverse sources of knowledge across many disciplines requires a shared "presentation" capacity-enabling the composition of apparently independent knowledge and expertise-which virtual universes provide.

These two results show how shared virtual presentation techniques and tools are much needed to support "de-fixation" and access to independent knowledge for the exploration of the unknown. This explains why virtual universes have a critical role to play. Not only do they contribute to the reduction of uncertainty through validation and optimization techniquescomputer-aided design (CAD) systems historically played this role by decreasing costs of experimentation-but, moreover, they offer a natural "compass" to orient and support the exploration of the unknown in all its forms, from new scientific phenomena to emerging technologies, novel uses and usages, and business models. These virtual universes for generativity should not be conflated with validation techniques-their value is much more in their capacity to generate surprising alternatives. They do so by connecting apparently independent dimensions and by helping users connect heterogeneous (independent) knowledge through new uses, technologies, complex systems, basic research, production, and creation. This phenomenon is already visible in aeronautics and the automotive industry, and is now spreading to all industries-most notably to life sciences, healthcare, construction, and services. Virtual universes contribute to support design logic, and they are the fuel of contemporary innovation.

A direct consequence is that, from a macroeconomic perspective, investments for shaping the unknown will develop in independent areas. For the design process, there are at least two well-known sources of independent knowledge: downstream users and upstream research.

On the one hand, virtual universes open up the possibility of much more integration for users and usages as an engine for exploring the unknown-not only by bringing knowledge from existing uses and users but also by enabling the creation of knowledge of alternative users and usages, individual and collective experiences, and emotions. Virtual universes strengthen the possibility of integrating the demand side into the design process. One could think of virtual universes as replacing the traditional proof of concept (POC), allowing the systematic generation of (virtual) prototypes at a very large scale to explore the multiple potential applications of generic technologies.

On the other hand, the value of investing in basic research might precisely come from the fact that basic research provides independent knowledge-knowledge that is neither the result of deductive problem-solving nor of optimized strategy! Basic research appears as a critical actor able to explore the unknowns of science-and, doing so, it brings back unexpected knowledge. The value of this knowledge is not in its applicability-this would correspond to dependent knowledge-but in its originality and unexpectedness. This mechanism is virtuous if two conflicting constraints are met. First, basic research has to be maintained independently from innovation, meaning it is neither application-driven nor problem-driven. Second, basic research should also be closely related to innovation so that 1) basic research knowledge can be used in innovation processes, and 2) innovation processes can provide basic research for new unknowns to avoid fixations by scientific communities and the laws of "publish or perish". Here again, virtual universes have a role to play: they can support basic research explorations, help identify basic research questions in innovation endeavors, and help import basic research results into innovation processes. Hence, even in basic research, virtual universes transform the scientific approach and support efficient exploration of the unknown.

Institutions to support shaping the unknown

It has been largely noticed that innovation requires a trusted environment in order to blossom and spread. 9 Institutions are needed to provide this trusted environment for shaping the unknown. Because they offer common reference points for groups of people, virtual twins act as new forms of institutions, creating the conditions for shared understanding, debate, and action. For instance, an infrastructure project related to new public transportation can be represented through a virtual twin of the city: the whole mobility system and related dimensions, such as building development policy, energy, and economic development on a given territory, can be modeled and simulated in a multi-factor approach. Third parties, such as contractors, local businesses, and administrations, can contribute by providing inputs and expressing constraints, and citizens can understand the project and contribute to design choices. When coherent with the logics of unknown exploration as explicated by design theory, virtual universes can be considered as assets with infinite value because not only are they non-rival goods, but also their value increases with usage.

For instance, the generative capacity of the virtual twin of a city can increase with the number of people accessing it and contributing to enriching its exploration paths. Virtual twins are, therefore, potentially "public goods". However, they are likely to be appropriated, for instance, if their creators or owners misuse them to bias explorations, fixate on certain paths, or hide exploration paths in the unknown. The concept of the unknown might thus require new protection mechanisms, such as a new legal status for "common and non-appropriable unknowns". Intellectual property might also be useful, probably in new forms, ensuring the publication of exploration paths and ensuring forms of recognition and rights for scouts, pioneers, or providers of the ways and means of unknown exploration. Patent law has evolved regularly over time to integrate new forms of inventiveness-new IP law might support the development of capabilities to explore the unknown and leverage the power of virtual twins. 10 The global response to the coronavirus disease 2019 (COVID-19) pandemic has shown how research could share vast amounts of data and intellectual property to accelerate the creation of knowledge on the disease and the discovery of new treatments. In this context of exploring the unknown, major research institutions offered a "no-fee, royaltyfree license" to their work involving the diagnosis and treatment of COVID-19 patients. Some initiatives were launched, such as the "Open COVID Pledge", urging researchers and companies to sign on "to make our intellectual property available free of charge for use in ending the COVID-19 pandemic and minimizing the impact of the disease."

The exploration of the unknown and the work in virtual universes are increasingly collective endeavors-with inevitable fixations and biases emerging either at the firm level or the ecosystem level. Appropriation, short-term profit, and shortsighted strategy can lead to severe pathologies and crises related to contemporary innovation. Examples include speculation bubbles on "killer applications" and "miracle technologies", forever technologies of the future, low success rate of market adoption, or even orphan innovation, i.e., situations where an innovation is expected by society, but companies fail to provide it. 11 Hence new institutional logics might be required, both at the firm level and the ecosystem level.

At the firm level, new governance principles might help protect and support firm capacities to shape the unknown. For instance, this has been one of the objectives of a new legal status for companies in France: the status of "profit-with-purpose company" protects and reinforces the capacity of the company to explore certain unknowns. 12 At the ecosystem level, researchers have identified the emergence of original institutions such as "colleges of the unknown" and "architects of the unknown". 13 These actors ensure that, in a given field of innovation, explorations are launched in all imaginable directions, are rigourously generated to avoid cognitive fixations, and cover a variety of alternatives with several time horizons. These actors also create a common language in the unknown, help measure and compare progress, and support coordination and interactions between designers. In a time of sustainable development goals (SDGs) and global transitioning to new technologies in areas such as energy and the digital economy, such organizations are very useful. However, not every self-appointed group can be a relevant college of the unknown. Virtual platforms could become game changers supporting efficient collaborative exploration and the development of quality criteria for the de-fixed, complete, and robust exploration of the unknown. 14 Quality criteria for collective exploration would systematize the identification of fixation at the ecosystem level and support the development of capacities to overcome those fixations.

Investing to build the "creation heritage" of future generations

Sustainability and other planetwide challenges are the domains demanding investment in the unknown today. The available solutions to face such contemporary threats are too limited, leading to unbearable sacrificial dilemmas such as agronomic pollutants vs. famine, carbonintensive energy vs. social riots, or, more recently, lockdown vs. epidemical diffusion. Going beyond sacrificial dilemmas is exactly the role of designers shaping the unknown. This requires huge investment-not only in intangible assets but also tangible assets, such as innovative long-term infrastructures for home improvement, mobility, cities, public health and care, etc. Digital also requires material infrastructure investment. The major question will thus be to orient rightly these investments towards challenges of the 21st century by correctly taking into account their intangible dimensions. Do these investments support unknown shaping, and are they overcoming collective fixations? Do they create long-lasting virtual assets able to capitalize on knowledge and know-how?

Moreover, how are these investments adapted to future generations? They should not only satisfy the predicted needs of future generations, but they should also provide future generations with the creative capacity and "creation heritage" to invent their own future. 15 Investing in virtual universes is a precondition to shaping the unknown and allows us to build a creation heritage. By this logic, investment in education is strongly needed as it is the key to unlocking these virtual universes and ensuring accessibility for the largest audience. From the perspective of investing in the capabilities for shaping the unknown, could there be anything more efficient than educating people to help de-fix themselves, develop capabilities to collectively and rigorously explore the unknown, and enable them to deal with virtual universes in a powerful and creative way? Perhaps the priority in terms of financing innovation today should be to invest in a collective culture of design based on shaping the unknown with virtual universes.