Vladimír Petrík

Makarand Tapaswi

Ivan Laptev
email: ivan.laptev@inria.fr

Josef Šivic
email: josef.sivic@cvut.cz

Learning Object Manipulation Skills via Approximate State Estimation from Real Videos

Keywords: Learning from videos, coarse 3D state estimation, imitation learning

Humans are adept at learning new tasks by watching a few instructional videos. On the other hand, robots that learn new actions either require a lot of effort through trial and error, or use expert demonstrations that are challenging to obtain. In this paper, we explore a method that facilitates learning object manipulation skills directly from videos. Leveraging recent advances in 2D visual recognition and differentiable rendering, we develop an optimization based method to estimate a coarse 3D state representation for the hand and the manipulated object(s) without requiring any supervision. We use these trajectories as dense rewards for an agent that learns to mimic them through reinforcement learning. We evaluate our method on simple single-and two-object actions from the Something-Something dataset. Our approach allows an agent to learn actions from single videos, while watching multiple demonstrations makes the policy more robust. We show that policies learned in a simulated environment can be easily transferred to a real robot.

Imagine that you want to assemble your bicycle. Humans are able to watch and learn from other people that demonstrate how to perform these actions, e.g., by watching videos from YouTube [START_REF] Miech | HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips[END_REF]. We are interested in providing intelligent agents the ability to learn object manipulation skills by watching videos (see Fig. 1).

Learning from demonstrations [START_REF] Argall | A survey of robot learning from demonstration[END_REF] aims at teaching robots to perform various actions based on demonstrations, often performed by experts. While previous works in this area have required humans to teleoperate the robot [START_REF] Zhang | Deep imitation learning for complex manipulation tasks from virtual reality teleoperation[END_REF], we wish to address a setting where the robot learns directly from video demonstrations. In particular, we are interested in understanding how to bridge the gap between the observed moving pixels (videos) and instructions that a robot can understand and execute. We address this question in two steps (see Fig. 2). First, building upon advances in computer vision and differentiable rendering, we propose Real2Sim, a method that lifts real world 2D videos to an approximate 3D state space representation of the hand and the manipulated objects (Sec. 3). Second, we use these automatically extracted trajectories along with reinforcement learning (RL) to learn policies that execute the actions in a 3D simulation environment corresponding to the real robot set-up and on a real robot (Sec. 4).

While there have been recent works on estimating detailed 3D meshes for hands and manipulated objects [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF], these methods do not generalize well to out-of-domain videos. Instead, we pursue an alternative solution: we extract only a coarse representation of the scene where we approximate the hand as a cylinder and objects as cuboids. Leveraging 2D hand-object detectors [START_REF] Shan | Understanding Human Hands in Contact at Internet Scale[END_REF] and pixelsegmentation methods [START_REF] He | Mask R-CNN[END_REF], we are able to reconstruct a coarse 3D state representation that shares perceptual similarities with the video. Even though the estimated states lack details (e.g. grasping an object), the coarse states of the dynamic scene act as a guide (a dense reward) for RL in a simulated environment, that in turn resembles the real world sufficiently allowing to transfer the learned policy to the real world. Our intermediate state representation allows a tangible and interpretable parsing of the video, and is specially effective at modeling actions that include object motion.

We investigate the following three questions in this work. First, we are interested in studying whether we can learn to perform simple object manipulations based on a single demonstration. We evaluate our approach on 9 simple object manipulation actions from the Something-Something action recognition dataset [START_REF] Goyal | The "something something" video database for learning and evaluating visual common sense[END_REF] and show that some clean videos are indeed capable of teaching an agent how to perform the task. Second, we hypothesize and empirically show that a robust policy can be trained by watching a handful of videos (a few-shot setting), thus reducing the impact of variable quality of individual demonstrations (Sec. 5). Finally, we are interested in studying how the object size or the initial gripper position impacts performance. We show that automatic domain randomization [START_REF] Akkaya | Solving rubik's cube with a robot hand[END_REF] presents a form of curriculum learning strategy that allows the agent to learn the task with gradually increasing complexity. We propose a challenging benchmark over the 9 actions, with randomized object sizes and initial gripper positions. Our action-specific metrics analyze whether the robot is able to perform the action correctly. We also demonstrate how the learned policies can be transferred to a real robot. We will make the code and data publicly available at https://git.io/JTPkj.

Related Work

We discuss recent advances in extracting 3D state/mesh representations from 2D images or videos and learning from video demonstrations.

Differentiable rendering. Understanding the 3D world in a projected image is a classical computer vision problem [START_REF] Roberts | Machine perception of three-dimensional solids[END_REF][START_REF] Gupta | Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics[END_REF]. However, obtaining ground-truth labels for 3D meshes is much harder than 2D object bounding boxes leading to challenges in 3D modeling [START_REF] Gkioxari | Mesh R-CNN[END_REF]. One approach is to learn a de-renderer [START_REF] Wu | Neural scene de-rendering[END_REF], an encoder that attempts to predict a structured scene representation, which can reconstruct the original scene through a graphics engine. This is extended to study object dynamics through visual de-animation [START_REF] Wu | Learning to see physics via visual de-animation[END_REF]. Alternatively, there have been attempts to reconstruct 3D object meshes by incorporating graph neural networks [START_REF] Wang | Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images[END_REF] or intermediate voxel spaces [START_REF] Gkioxari | Mesh R-CNN[END_REF]. Recent developments in differentiable neural rendering [START_REF] Kato | Neural 3D Mesh Renderer[END_REF][START_REF] Liu | Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning[END_REF], have enabled approaches that estimate both the 3D texture and shape [START_REF] Chen | Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer[END_REF] while only requiring 2D supervision (e.g. a segmentation mask). Ours is a parameter-free method that can estimate an approximate 3D representation from a single video.

Hands and objects in 3D. Joint understanding of hands and objects has implications from action recognition [START_REF] Damen | The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines[END_REF][START_REF] Tekin | H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions[END_REF] to virtual or agumented reality [START_REF] Höll | Efficient Physics-Based Implementation for Realistic Hand-Object Interaction in Virtual Reality[END_REF][START_REF] Tang | GrabAR: Occlusion-aware Grabbing Virtual Objects in AR[END_REF]. Inspired by the SMPL person model [START_REF] Bogo | Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image[END_REF], Romero et al. [START_REF] Romero | Embodied Hands: Modeling and Capturing Hands and Bodies Together[END_REF] develop MANO, an articulated hand model has helped researchers focus on studying various hand deformations [START_REF] Zhou | Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data[END_REF]. Recently, there have been large data collection efforts for the joint study of hands and objects: as 2D bounding boxes [START_REF] Shan | Understanding Human Hands in Contact at Internet Scale[END_REF][START_REF] Materzynska | Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks[END_REF], or 3D pose estimates [START_REF] Hampali | HOnnotate: A method for 3D Annotation of Hand and Object Poses[END_REF]. Nevertheless, joint 3D reconstruction of hand and object, especially during manipulation remains a challenging problem. There have been some efforts in this direction, such as leveraging synthetic datasets and utilizing manipulation constraints [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF], or addressing the challenges in 3D annotation through temporal consistency in a video [START_REF] Hasson | Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction[END_REF]. In this work, we propose an alternative solution. We hypothesize that a coarse 3D state representation is sufficient to teach a robot to perform these actions, and propose an optimization procedure to estimate hand-object trajectories from monocular videos.

Learning from demonstrations. Enabling a robot to learn policies based on expert demonstrations is a well-studied problem [START_REF] Argall | A survey of robot learning from demonstration[END_REF][START_REF] Hazara | Reinforcement learning for improving imitated in-contact skills[END_REF][START_REF] Vecerik | Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards[END_REF]. Learning from demonstrations (LfD) circumvents designing task-specific reward functions [START_REF] Yu | Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning[END_REF] that are often challenging to devise. Many works in this direction leverage humans teleoperating a robot, e.g. [START_REF] Zhang | Deep imitation learning for complex manipulation tasks from virtual reality teleoperation[END_REF][START_REF] Caccavale | Kinesthetic teaching and attentional supervision of structured tasks in human-robot interaction[END_REF][START_REF] Strudel | Learning to combine primitive skills: A step towards versatile robotic manipulation[END_REF]. In contrast, we are interested in leveraging the large diversity of videos where people perform various object manipulations as our demonstrations.

Learning from videos include directions such as estimating perceptual reward functions from a small number of demonstrations [START_REF] Sermanet | Unsupervised Perceptual Rewards for Imitation Learning[END_REF]; learning a visual representation via multiple views and metric learning, followed by learning a policy using a single third person demonstration [START_REF] Sermanet | Time-Contrastive Networks: Self-Supervised Learning from Video[END_REF]; imitation from observation by learning to predict different viewpoints [START_REF] Liu | Imitation from Observation: Learning to imitate behaviors from raw video via context translation[END_REF]; or very recently, learning pixel-level translation to convert human demonstration to the robot's perspective [START_REF] Smith | AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos[END_REF]. Our goal is similar, but our approach differs. Without the need for multiple views, we estimate a 3D physical state representation through an optimization algorithm, thus mapping the video to a simulated environment. Parallel to Our input is a few video demonstrations for each action, here depicting pull left to right. Real2Sim (center): Our optimization based approach obtains coarse 3D state representations for the hand (cylinder) and object (cuboid) based on a combination of physical losses modeling hand-object interactions and perceptual losses measuring similarity between the rendered image and a segmentation mask. Multi-sequence learning (right): I. Our approach estimates several trajectories, one for each video. II. We align these 3D trajectories in space and time. III. Environments with random object size and starting hand position are sampled. IV. The learning policy generates a simulated hand-object trajectory. V. We compute a dense reward that trains the agent to mimic the estimated trajectories.

our work, Shao et al. [START_REF] Shao | Concept2Robot: Learning Manipulation Concepts from Instructions and Human Demonstrations[END_REF] present Concept2Robot that leverages an action classifier as a reward signal. They learn a single multi-task policy with a natural language instruction encoder that generalizes to small variations in actions. Nevertheless, the limited performance of the classifier, especially on simulated videos leads to noisy reward signals.

Mimicking actions. 3D human reconstruction [START_REF] Kanazawa | End-to-end Recovery of Human Shape and Pose[END_REF] has seen use in teaching robots to mimic human actions [START_REF] Peng | DeepMimic: Example-guided Deep Reinforcement Learning of Physics-based Character Skills[END_REF][START_REF] Peng | SFV: Reinforcement Learning of Physical Skills from Videos[END_REF][START_REF] Peng | Learning agile robotic locomotion skills by imitating animals[END_REF]. While human pose can be extracted from a video with surprising accuracy, this is currently not the case for estimating the pose of 3D objects manipulated by hands. Our approximate scene representation addresses these challenges as it consists of coarse 3D blocks representing the main objects (and the hand) in the input video. In addition, our videos/demonstrations are aligned automatically by leveraging components of the state representation.

Real2Sim: Approximate State Estimation from Video

We model the structure and motion of the hand and object(s) in the video with coarse 3D models: a hand (wrist onwards) is represented as a cylinder with a fixed height and radius approximating the dimensions of a real hand, while objects are approximated as cuboids. In this section, we present our approach for 2D spatio-temporal video parsing, define the state space, and design perceptual and physics-based losses that optimize and estimate the states for a video.

Video pre-processing. We parse the video demonstrations using 2D visual reasoning tools: Mask-RCNN [START_REF] He | Mask R-CNN[END_REF] for segmentation masks, and Hand-Object detector [START_REF] Shan | Understanding Human Hands in Contact at Internet Scale[END_REF] that also provides a binary output to indicate touching or not. To ablate detection errors, we use ground-truth boxes by [START_REF] Materzynska | Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks[END_REF]. The final outputs of this module, obtained through a Kalman filter, include frame-level bounding boxes and segmentation masks for the hand and object(s). Please refer to the Appendix A.1 for details.

Approximate 3D state space

Consider an input video v = (v 1 , . . . , v T ) with T frames, demonstrating how to perform some action, e.g. pull left to right (see Fig. 2). We define a state space to model the hand-object interaction, that mimics the visual content. (i) We assume the camera is fixed for each video and localize it in 3D using distance to origin, azimuth, and elevation. (ii) The hand is modeled as a cylinder (40 mm radius and 150 mm height), with one of the edges symbolizing the manipulator/fingers. We use a 5D representation: h = (h p , h θ ), where h p encodes the 3D position in Cartesian space and h θ captures two rotations: azimuth and elevation. (iii) Each object, represented as cuboid, is encoded by a 9D vector o = (o p , o s , o θ ) corresponding to 3D position in Cartesian coordinates o p , 3D object size o s , and the angle-axis formulation for the object rotation o θ . (iv) Finally, we also encode whether the hand is touching the object as a binary label τ for each frame. This leads to a D = 18 dimensional state representation (camera: 3, hand: 5, object: 9, touch: 1) for videos/actions with one object.

Losses and optimization

Our goal is to estimate a 3D state trajectory in R T ×D over the T video frames that mimics the visual content of the demonstration. We employ two categories of loss functions, perceptual and physical. While the general problem of 2D to 3D has multiple solutions, two factors work in our favor: a fixed hand size helps estimate the distance between the camera and the hand; and a non-zero camera elevation that mimics ego-views helps estimates the depth of the objects placed on the surface.

Perceptual loss aims to estimate a 3D state trajectory that closely resembles, when projected, the video. We achieve this by representing the hand (cylinder) and objects (cuboids) as triangular meshes [42] that are rendered through a differentiable neural renderer [START_REF] Kato | Neural 3D Mesh Renderer[END_REF], projecting the 3D state space of the hand and object to an image. We render binary object silhouettes without textures as they can be easily compared against the binary segmentation masks extracted from the video. As the optimization uses predicted segmentation masks, it can be considered as a self-supervised approach. The perceptual loss measures the reprojection error summed over all frames of the video for the hand and object(s) of interest in the scene:

L perceptual = T t=1 1(m h t ) • r h t -m h t 2 + i 1(m oi t ) • r oi t -m oi t 2 , (1)
where m h t , m oi t correspond to the predicted hand and ith object segmentation masks and r h t , r oi t correspond to rendered outputs. As frame segmentation or tracking may be unreliable, we make the estimation robust by applying the loss only when the mask is non-zero, indicated as 1(•). Additionally, a few missed detections are also interpolated using physics-based losses described next.

Physics based losses provide regularization and model the hand-object interaction. As part of the regularization, we minimize the acceleration (double-derivative in time) of all the object positions öp and rotations öθ , and the hand position ḧp and rotation ḧθ :

L acc = ḧp + ḧθ + i öi p + öi θ . (2)
Acceleration loss smooths the trajectory and approximates the law of conservation of momentum. Additionally, as the object is assumed to be non-deformable, we also minimize the distance between the object size o s and its mean over time ōs , L size = i t o i s (t) -ōi s . Finally, we regularize the hand-object interaction by imitating the law of inertia with infinite friction. We assume that only one object is manipulated at any time, and encourage the model to position the hand to be closer to the object if it is moving, and to minimize the object velocity otherwise:

L interact = p ho • ȯp -ḣp + (1 -p ho ) • ȯp , (3)
where

p ho = 1 -σ( h p -o p )
is the probability that the hand is touching an object based on their positions, and σ() is a parameterized sigmoid function that takes in a positive distance value and produces outputs in the range [0, 1]. The first term ensures that the hand and object move together when the hand touches the object and the second term penalizes object motion when the hand is not touching the object. Hand-object interactions are also encoded by the touch indicator τ predicted by [START_REF] Shan | Understanding Human Hands in Contact at Internet Scale[END_REF]. For the frames where the hand is said to touch the object, we set h p = o p .

Optimization. We use gradient descent to estimate the 3D state space trajectory that minimizes the total loss, a weighted combination of all terms:

L = w p L perceptual + w a L acc + w s L size + w i L interact .
Owing to the differential neural renderer, we are able to backpropagate through the perceptual loss, while all other terms use standard differentiable components. See Appendix A.2 for details.

Learning Object Manipulation Policy from Multiple Videos

The objective of RL is to learn to manipulate object(s) so as to imitate the trajectories extracted from the videos. We use these trajectories to construct a reward function that trains the policy in a physics-based simulator mimicking the real world. To learn from multiple videos, trajectories are first spatio-temporally aligned. Note that, across all actions, we use the same reward function that encourages the policy to learn to mimic hand/object trajectories. This is a key aspect of our work as it does not require handcrafting a new reward for each action. Fig. 2 (right) presents an overview.

Spatio-temporal alignment of multiple extracted trajectories

Prior to using the trajectories from multiple demonstrations as part of the reward, we spatially align all trajectories of the same action by compensating for the camera orientation and the initial object position. After the alignment, all trajectories are expressed in the same reference frame given by the starting object position. In the case of multi-object actions we used the non-manipulated object for the alignment (see Fig. 6 in Appendix A.3).

The trajectories are also re-scaled in time such that all of them have the same duration and the action starts at a fixed timestamp. We estimate the action phase, the temporal part of the video where the actual action is performed by analyzing if the object is in motion (velocity ȯp above threshold) or if the hand is touching the object (predicted from [START_REF] Shan | Understanding Human Hands in Contact at Internet Scale[END_REF]). An approach phase inserted at the beginning gives the gripper time to move to its initial position to execute the action. A leave phase at the end requires the gripper to be opened if the object should not remain grasped and the final position is set to be above the object position from the trajectory.

Simulator and learning setup

The simulated environment models the target robot environment that consists of a parallel jaw robotic gripper and cylindrical objects. The gripper pose is controlled by linear and angular velocity and the gripper closing is specified by the gap between gripper fingers. The simulation is performed for a fixed maximum horizon that corresponds to 10 s of simulated time and is prematurely reset if the gripper hits the ground. When the environment resets, the poses of the hand and object(s) could be set according to the first state of the trajectory. However, a policy learned with this constant starting state generalizes poorly to other starting conditions. Therefore, we randomize the gripper starting poses and the object sizes at the beginning of each episode. In particular, we employ automatic domain randomization, explained later in Sec. 5.2.

The goal of RL is to find a feed-forward policy that maps the observation to the distribution of actions that control the gripper. We represent the policy π as a Gaussian distribution: π(a|s) = N (µ(s), Σ), where a ∈ R 7 is the gripper command, s ∈ R 8 is the state containing time (1D) and gripper information (6D pose, 1D gripper opening), µ(s) ∈ R 7 is a mapping represented by a neural network (three hidden linear layers of size 128), and the diagonal covariance matrix Σ ∈ R 7×7 is independent of the states. The Σ parameters and the neural network are trained by Proximal Policy Optimization [START_REF] Schulman | Proximal Policy Optimization Algorithms[END_REF] to maximize the expected discounted return defined as

R = E H i=0 γ i r(t i ) , (4)
where t i is timestamp at step i, horizon H specifies the maximum number of time steps, r is the immediate reward that encodes the distance to trajectories extracted from the videos (presented next), and γ is the discount factor, set to 1 for our task to assign equal importance to future rewards.

Learning a policy from single or multiple videos

We start by defining the immediate reward for a single video as it forms the basis of multi-video learning. Note that such policies are often brittle as small errors in state estimation can lead to irrecoverable errors in the policy. For example, for the push action, we expect the hand to be behind the object. However, if this is not respected in the estimated trajectory, RL cannot recover from this error. For learning from a single video, the immediate reward r v at time t is defined as:

r v (t) = q∈{hp,h θ ,o i p ,o i θ ,τ } w q exp - 1 2 l q d q (q(t), q(t)) , (5)
where q is one of the quantities (hand position, object orientation, etc.) extracted from the video, q is the corresponding quantity computed from the simulated environment, function d(•, •) measures distance between those quantities, l q are constant length-scales used to compensate for differences in units, and w q are constant weights used to adjust the importance of different quantities. We assign the highest weight to object position as it is a crucial factor in judging the success of an action, and is difficult to learn as the policy may first need to learn other skills such as gripping the object.

We use different distance functions d q to compare simulated and estimated trajectories: (i) squared euclidean norm for hand and object positions; (ii) squared angular distance for hand azimuth and elevation; (iii) squared angular distance between quaternions for object orientations; and (iv) the following weak signal to learn gripper closing:

d(τ, τ ) = τ -τ 2 if reference touch signal detected in the video, i.e. τ = 1, ∞ otherwise . (6)
This ensures that a positive reward is provided for closing the gripper only when the hand is touching the object, and the robot may arbitrarily choose to open/close the gripper at other timesteps.

Reward for multiple videos. To learn a robust policy unaffected by errors in trajectory estimation, we propose to leverage multiple videos of the same action. We train on S a , the subset of demonstrations corresponding to an action a, by averaging aforementioned immediate reward for a single video (5):

r a (t) = 1 |S a | v∈Sa r v (t) . (7)
Note, that the average is computed over exponential rewards making it robust to outliers. Even when there are multiple ways to achieve the same task, our policy learns to mimic the most common trajectory and not the averaged trajectory. Rewards r v (Eq. 5) and r a (Eq. 7) are used as the immediate reward in Eq. ( 4) that is maximized by RL.

Evaluation

We first present an overview on the set of actions, a simulator-based benchmark, and metrics to evaluate the performance of learned policies. We then present an ablation study highlighting the impact of visual recognition tools and a curriculum learning approach. Finally, we compare policies learned on single or multiple videos and conclude with a brief summary of the real world robot setup.

Evaluation setup

Actions. We evaluate our approach on nine actions from the Something-Something action recognition dataset [START_REF] Goyal | The "something something" video database for learning and evaluating visual common sense[END_REF]: five single object manipulation tasks (push/pull/pick) and four two-object manipulation tasks (put). Fig. 3 illustrates one video per action category. For each action, six diverse videos in terms of backgrounds and manipulated object categories and shapes are selected.

Benchmark and metrics. We evaluate the robustness of trained policies by creating a benchmark of 1,000 samples that includes a variety of starting poses for the hand/gripper and sizes for the object(s). This includes challenging setups that are not similar to video demonstrations, e.g. the hand starts behind the object. We also create an easier benchmark with 1,000 samples drawn from a limited set of starting poses. All results are presented on the hard benchmark unless stated otherwise.

An estimated reward function is a proxy for the metric and is not always correlated with the successful execution of the action. Therefore, we use hand designed action-specific metrics (only during evaluation) that analyze the state of the hand and object while executing the action. Our stringent metric indicates a binary success/failure for each execution, e.g. pull/push actions check that the hand is in the correct position during object motion, that the object is standing, and it has moved at least 50 mm in the correct direction. Additional details are in Appendix B.

Experiments

Real2Sim setups. We analyze the impact of failures in visual parsing and obtain 3D state trajectories in three ways: A. ground-truth (GT) object/hand boxes [START_REF] Materzynska | Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks[END_REF] and GT action phase localization; B. GT boxes with predicted action phase; and C. predicted boxes with predicted action phase. Note that all setups still use predicted segmentation masks and automatic tracking. As the videos are crowdsourced for action recognition, there is no GT for the 3D states. Thus, we use the action-specific metrics defined above to also compare the different Real2Sim setups. The average success rate over all 54 videos for the three setups above is: A. 67%; B. 72%; and C. 61%. Surprisingly, method B works best, possibly due to the trajectory being estimated for the complete video followed by action phase localization. Nevertheless, as we will see later, policies learn actions best with setup A. Additional qualitative and quantitative analysis can be found in Appendix C.

Curriculum learning. We wish to learn policies that are robust to the initial gripper pose. This is normally achieved by randomizing the starting gripper position and orientation during training, i.e., h p ∼ N (0, σ 2 ) , h θ ∼ U(-180°, 180°), where N and U are Normal and Uniform distributions, and σ is set to 250 mm. However, the blue bars in Fig. 4 show that the performance drops as σ increases indicating that learning with a randomized gripper pose is challenging. Inspired by Automatic Domain Randomization (ADR) [START_REF] Akkaya | Solving rubik's cube with a robot hand[END_REF], we design a curriculum learning strategy that linearly increases the randomness of the gripper pose during training. Policies that use ADR for both the 3D position and orientation (Fig. 4 Evaluating policies trained on different Real2Sim setups. Fig. 5 shows the success rate for each action for policies trained on multiple videos evaluated on the easy (top) and hard (bottom) benchmarks. While states predicted using method A (GT boxes and action phase, blue bars) result in higher performance, states that use method C (predicted boxes and action phase, green bars) degrade in performance by 10-15% on both benchmarks. Although method C results in higher performance for some actions (e.g. put in front of ), on average, the estimated vision works slightly worse than ground-truth as expected. However, note that the action put onto is found to be particularly challenging as it requires precise modeling of objects, and about 5-7% of the gap between methods A and C is explained by this action alone. Overall, the trained policies are successful at executing the actions for a majority of initial states on both benchmarks. The gap between easy and hard benchmarks is also about 10-15%, indicating the challenges in starting position and strictness of our metric. A common failure case for our policy is when the gripper and object start close to each other and the gripper collides with the object while re-positioning in the approach phase.

Learning from single vs. multiple videos. We demonstrate the benefits of learning a policy from multiple videos by comparing them against policies trained on individual videos for all actions. In addition, we compare the proposed multi-video reward function to a baseline [START_REF] Peng | DeepMimic: Example-guided Deep Reinforcement Learning of Physics-based Character Skills[END_REF] that computes the reward by maximizing over the trajectories (instead of sum, see Eq. ( 7)). Table 1 shows that learning from single videos leads to a high variance in success rate due to the quality of the demonstration and state estimation process. In contrast, our proposed approach learns the action even if only a few trajectories are correct (see put behind/in front of ). Finally, the multi-video policy used in the baseline (DeepMimic [START_REF] Peng | DeepMimic: Example-guided Deep Reinforcement Learning of Physics-based Character Skills[END_REF]) results in lower performance than our approach on most actions. The baseline computes maximum across trajectories, essentially selecting the easiest trajectory that would we consider whether the object is in motion, by checking if the velocity is greater than 70 mm s -1 . These predictions are quite reliable with an average intersection-over-union of 0.81.

A.3 Learning object manipulation policy from multiple videos

Spatial alignment. We spatially align all trajectories of the same action by compensating for the camera orientation and the initial object position as shown in Fig. 6. The spatial alignment merely resets the frame-of-reference (camera and initial object position) to an object in the scene, something that can be achieved for most actions. Reinforcement learning parameters. Reward weights required for computation of Eq. ( 5) are shown in Table 2. We use PPO [START_REF] Schulman | Proximal Policy Optimization Algorithms[END_REF] algorithm that collects 70 episodes per policy update. Policy update uses learning rate 3 • 10 -4 and is performed for 25 epochs, value 0.2 for likelihood ratio clipping threshold, value 100 for gradient clipping, weight 1 • 10 -8 for entropy loss and generalized advantage estimation with lambda set to 0.95. Discount factor is set to 1. Total number of policy updates is set to 1,000.

For Automatic Domain Randomization (ADR), we linearly increases randomness for gripper orientation from the 1st until the 400th policy update and for gripper position from 500th until the 900th policy update. The policy is represented by a fully connected neural network with three hidden layers each containing 128 neurons. The models are implemented using PyTorch and rlpyt, a RL library for PyTorch.

Parameter h p h θ o i p o i θ τ w q 0.2 0.2 0.5 0.05 0.05 l q 100 10 100 10 10000

B Benchmark and Metrics

The easy and hard benchmarks are sampled randomly from distributions specified in Table 3. The random sample is discarded if a collision is detected among any two of hand, object(s), or ground plane. In total 1,000 collision free samples are obtained for each benchmark and used for the evaluation.

Metrics. We design an evaluation metric for each action separately by analyzing the positions and orientations of the hand and object trajectories. For pull and push actions, the metric requires that object is moved in the correct direction by at least 50 mm. During the object motion, the hand has to be oriented in the direction of the motion for push and in a reversed direction for pull actions (see Fig. 7). Note that we also require the object stay upright (not fall) during the pull or push actions. For pick something up (not visualized), the metric checks that the object is held at least 10 mm above the ground by the gripper. Finally, for the two object actions (put), we consider the final state of the objects. The final position of the manipulated object should be within a 120°arc around the static object, and within 500 mm radius from the center of the static object. The proposed metric is binary, and evaluates whether the policy executes the action for a given starting hand position and object size. Visualization of the metrics used to evaluate whether the action is performed correctly. Note how the hand is in the direction of motion for pull actions, while in the opposite area for the push actions. Two object actions allow the object to be put in a specific cone of area behind/in front of/next to the object. For put onto, we show the front view, indicating that the second object should be placed above the static one. Please refer to the text for exact numerical details.

C Additional Results: Real2Sim

Quantitative results for different ablation methods for Real2Sim were presented as summaries in the main paper. We elaborate on those in Table 4, showing the fraction of correctly extracted trajectories for each action and method. Recall, we analyze performance based on three methods depending on whether automatic or ground-truth was used for performing spatio-temporal localization: A. ground-truth object/hand boxes and ground-truth action phase localization; B. ground-truth boxes with predicted action phase; and C. predicted boxes with predicted action phase. We observe that method B works best, possibly due to the trajectory being estimated for the complete video followed by truncation of states using predicted action phase localization. In contrast, for method A, we first truncate the video and then perform state estimation. Visualization. The metrics provide a limited understanding of the state estimation process (see Sec. 5.2). We visualize the estimated hand and object trajectories from a top-view perspective in Fig. 8.

Method

To obtain better insights, we visualize additional results as GIFs in an HTML page real2sim.html showing all 6 videos for each of the 9 actions. In each row of the table, we show the full video, followed by segmentation masks m h t , m oi t and corresponding 3D state renderings via the neural renderer r h t , r oi t . All video ids correspond to the original ids in the Something-Something dataset. The estimations from all three methods are shown as three columns of the table.

D Policy Training Results

We present additional results and analysis for training a policy with different settings.

D.1 Curriculum learning

In the main paper, we looked at how curriculum learning with Automatic Domain Randomization (ADR) helped our agent to learn a better policy. While the main paper presented results on the hard benchmark (see Fig. 4), here, we present additional results on the easy benchmark in Fig. 9. In particular, we observe the same trends. The blue bars in Fig. 9 show that the performance drops as σ increases indicating that learning with a randomized gripper pose is challenging. Our strategy to use ADR for both the 3D position and orientation (Fig. 9 9: Success rate for multi-video policies trained with different randomization strategies using state trajectories from method A, and evaluated on the easy benchmark. Legend: the first number represents variability in the 3D position of the gripper, σ in mm; the second corresponds to the range of hand orientation in degrees. Five policies are trained for each parameter and we report mean and standard deviation (error bar in the plot).

D.2 Impact of GT in Real2Sim setups

We present additional analysis of the difference in performance due to the use of GT bounding boxes or GT action phase. Recall that these are referred to as method A, B, and C in the main paper. On average, Fig. 5 shows that the estimated vision works slightly worse than ground-truth as expected. Actions put behind and put onto follow this trend. For put next to, the performance is within error bars. By looking at real2sim.html (supplementary), we see that there are a few demonstrations where Real2Sim could be improved with better object segmentation and tracking. We think these individual instances are outliers and are not representative of estimated vision working better, even for put in front of. Referring to real2sim.html method A, we see that video 5 is challenging due to the elongated scissors, while video 6 fails as the object is placed too far for put next to. For put in front of, video 1 places the object too far, and video 2 fails probably due to the GT box extending beyond the object mask.

Big performance gap for put onto. We observe that method A achieves 60% success rate for put onto, while methods B and C are close to 0%. This can be explained by analyzing the GIFs for video 2 and video 4 in real2sim.html, where states estimated by method A are better than method C. Quantitatively, method A gets 2/6 correct, while method C gets none (Table 4). RL is able to use these 2 trajectories to learn the task. Having ground-truth 2D boxes alleviates significant confusion while tracking the two objects. Additionally, the big gap is also likely due to the binary nature of the metric that requires balancing objects: it either works, or the object falls, there is no half-way success.

D.3 Learning from single vs. multiple videos.

Similar to Table 1 of the main paper that shows the performance of learning from single vs. multiple videos on the hard benchmark with states obtained from method C, we present quantitative performance in a few more different settings.

Visualization. Similar to the results for Real2Sim, we create another HTML page, policies.html showing results of trained policies. For each action, we present one successful and one failure example in the simulator, and the result of transferring the policy to a real robot.

Figure 1 :

 1 Figure 1: Our robot watches a few video demonstrations and learns to perform the observed action by estimating a coarse hand-object 3D state.

*

  indicates equal contribution 4th Conference on Robot Learning (CoRL 2020), Cambridge MA, USA.

  arXiv:2011.06813v1 [cs.RO] 13 Nov 2020

Figure 2 :

 2 Figure2: Overview of our method. Demonstrations (left): Our input is a few video demonstrations for each action, here depicting pull left to right. Real2Sim (center): Our optimization based approach obtains coarse 3D state representations for the hand (cylinder) and object (cuboid) based on a combination of physical losses modeling hand-object interactions and perceptual losses measuring similarity between the rendered image and a segmentation mask. Multi-sequence learning (right): I. Our approach estimates several trajectories, one for each video. II. We align these 3D trajectories in space and time. III. Environments with random object size and starting hand position are sampled. IV. The learning policy generates a simulated hand-object trajectory. V. We compute a dense reward that trains the agent to mimic the estimated trajectories.

Figure 3 :

 3 Figure 3: Example video demonstrations for the 9 actions used to evaluate our approach. Three frames for each video depict the beginning, middle, and the end of the annotated action phase.

Figure 4 :

 4 Figure4: Success rate for multi-video policies trained with different randomization strategies using state trajectories estimated from GT boxes/GT action phase. Legend: the first number represents variability in the 3D position of the gripper, σ in mm; the second corresponds to the range of hand orientation in degrees. Five policies are trained for each parameter and we report mean and standard deviation (error bar in the plot).

Figure 6 :

 6 Figure 6: Top-view explaining the spatial alignment of extracted trajectories for the action pull left to right. Time is indicated by transparency, the most transparent color represents the start of the trajectory. For one video, (a) shows the estimated camera pose, hand (orange), and object (green) trajectories, while (b) shows the trajectories after alignment, compensating for camera pose and initial object position. (c) depicts object trajectories extracted from multiple videos of the same action, and (d) presents their aligned version.

Figure 7 :

 7 Figure7: Visualization of the metrics used to evaluate whether the action is performed correctly. Note how the hand is in the direction of motion for pull actions, while in the opposite area for the push actions. Two object actions allow the object to be put in a specific cone of area behind/in front of/next to the object. For put onto, we show the front view, indicating that the second object should be placed above the static one. Please refer to the text for exact numerical details.

  orange) achieve the best results and are used in all other experiments.

	Success rate	0.2 0.4 0.6 0.8 1.0 0.0	Pull left to right	Pull right to left	Push left to right	Push right to left	Average	0/360 50/360 100/360 150/360 200/360 250/360 ADR/360 ADR/ADR position/angle

Table 2 :

 2 Reward parameters for RL.

Table 3 :

 3 Ranges for benchmark generation. Symbol U represents uniform distribution.

Table 4 :

 4 Benchmarking states.

  orange) achieves the best results and is used in all other experiments.

	Success rate	0.0 0.2 0.4 0.6 0.8 1.0	Pull left to right	Pull right to left	Push left to right	Push right to left	Average	0/360 50/360 100/360 150/360 200/360 250/360 adr/360 adr/adr
	Figure

Acknowledgment. This work was funded in part by the Louis Vuitton ENS Chair on Artificial Intelligence, the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute), and the European Regional Development Fund under the project IMPACT (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000468).

 5: Success rate for policies trained on multiple videos with trajectories estimated from different Real2Sim setups. Top: easy benchmark; Bottom: hard benchmark. Each bar represents mean and std-dev over ten policies. maximize reward. However, if this chosen trajectory does not successfully execute the action, it leads to negligible success rates. as is seen for the relatively easier actions of pull left to right/right to left. The success rate (in %) for single and multi-video policies and for the baseline [START_REF] Peng | DeepMimic: Example-guided Deep Reinforcement Learning of Physics-based Character Skills[END_REF]. Single video performances are sorted in descending order, and their average score is presented in the "average" column.

Results are on the hard benchmark with states from method C, predicted boxes and action phase. Please refer to Appendix D.3 for similar results on the easy benchmark, and with states estimated from method A.

Transferring learned skills to the real robot. As the trained policy predicts the linear and angular velocity and the amount of gripper opening based on the current state of the gripper, we can compute the whole execution trajectory offline for an object with known initial pose (required for the alignment, and assumed known for our setup). We use numerical inverse kinematics tracking to compute the robot joint trajectory from the gripper Cartesian trajectory and quantize the gripper opening/closing due to lack of real-time gripper control capability for our Franka Emika Panda robot, shown in Fig. 1. Please refer to the supplementary video for a demo.

Conclusion

We presented an approach to teach robotic agents simple object manipulation skills by watching a few videos. We proposed a method that estimates a coarse 3D state representation for the hand and object(s) through a combination of 2D visual recognition, differentiable rendering, and an optimization method that learns from perceptual and physics based losses. These approximate state trajectories are used in an RL setup to successfully learn object manipulations for 9 single-and multi-object actions. We performed a thorough evaluation in a simulated environment, highlighting the benefits of adopting a curriculum learning strategy and learning from multiple videos, and also showed that the learned policies can be transferred to a real robot. Interesting future directions include incorporating physics into the Real2Sim estimation and scaling up to more actions.

Appendix

We present additional details for our method and a variety of qualitative and quantitative results. This appendix is complemented by additional material including a video describing the contributions in brief and demonstrating results on a real robot, and two HTML files: real2sim.html and policies.html showcasing qualitative real2sim and policy results respectively.

A Implementation Details

We present some additional implementation details for (i) Video pre-processing that estimates hand and object detections and segmentation masks; (ii) Real2Sim, that estimates 3D state trajectories from videos; and (iii) RL that learns a policy to mimic object manipulation based on extracted trajectories.

A.1 Video pre-processing

We parse each video with a frame-level hand-object detector and semantic segmentation method.

(i) Mask-RCNN [START_REF] He | Mask R-CNN[END_REF] predicts segmentation masks for multiple COCO categories [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF], including the person label that segments the hand, and various objects that may appear in the video.

(ii) Hand-Object detector [START_REF] Shan | Understanding Human Hands in Contact at Internet Scale[END_REF] is a recent work that detects hands, objects, and their interactions (touching or not). As an alternative to automatic detection, we also analyze the impact of using ground-truth boxes (Something-Else [START_REF] Materzynska | Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks[END_REF]). However, note that this still uses automatic segmentation to convert boxes to pixel-level masks.

Tracking. We track the hand and object(s) of interest by adopting a Kalman-filter [START_REF] Bewley | Simple Online and Realtime Tracking[END_REF] that links frame-level detections using a simple intersection-over-union overlap based metric. Missed detections are ignored, and the segmentation mask is set to all zero. The physics-based losses, especially the acceleration loss, interpolates across missed detections by preventing rapid changes in position and orientation.

A.2 Real2Sim

3D state estimation parameters. States are represented as learnable parameters (real numbers) that are normalized through sigmoid or tanh functions.

We initialize 3D state parameters of the hand (3D position, elevation) and the object (3D size and 3D position) by sampling from a normal distribution: 0.1 • N (0, 1). Hand azimuth is initialized from 0.1 • N (π/2, 1) to model the hand orientation in egocentric videos (pointing away from the camera at 90°), and finally object rotation is initialized as 0.01 • N (0, 1) as we expect objects to be upright.

We normalize these parameters as follows: (i) object sizes are normalized through a sigmoid function in range 0 mm to 300 mm; (ii) object position is normalized through a tanh function in range -1200 mm to 1200 mm; (iii) hand position is normalized through a tanh function in range -1500 mm to 1500 mm; and (iv) hand elevation is normalized using a tanh function in range -90°to 90°. We found that normalizing hand azimuth makes state estimation challenging, possibly due to 0°and 360°c orresponding to the same angle. The object position in z direction is set to half the object size in z direction as objects may lie on the table.

Weights for the loss terms are as follows. For the perceptual loss w p = 0.3, acceleration term w a = 5, interaction term w i = 1, and object size term w s = 1000. Note that the high weight w s ensures that the object size does not change over the estimated trajectory.

Learning details. We use the Adam optimizer with a (considerably high) learning rate 10 -2 to update the state parameters. All states for the entire video are updated at each iteration as this allows imposing acceleration based regularization at every frame. The parameters are updated for 400 iterations (most parameters converge typically around 200-250 iterations). The models are implemented using PyTorch.

Action phase estimation. Given the entire 3D state trajectory, we use two cues to predict when the action is taking place. First, we consider whether the hand is touching the object [START_REF] Shan | Understanding Human Hands in Contact at Internet Scale[END_REF]. Second, In particular, Table 5 shows the performance on the easy benchmark when using states estimated from method A. Our agent is able to learn to perform most actions, often with success rates greater than 90% (5/9 actions). Table 6 presents the results on using the same states, but on the hard benchmark. We notice a drop in performance for all actions, particularly for starting positions of the hand being away from the camera. Finally, Table 7 showcases the performance when using states estimated from method C on the easy benchmark.

Overall, we observe that our proposed multi-video learning method outperforms learning from single videos, as some demonstrations may be noisy. In addition, using sum (see Eq. ( 7)) instead of max [START_REF] Peng | DeepMimic: Example-guided Deep Reinforcement Learning of Physics-based Character Skills[END_REF] shows consistent performance improvements across all experiments. The success rate (in %) for single and multi-video policies and for the baseline [START_REF] Peng | DeepMimic: Example-guided Deep Reinforcement Learning of Physics-based Character Skills[END_REF]. Single video performances are averaged in the "average" column. Results are on the easy benchmark with states from method C.