--- license: apache-2.0 library_name: diffusers pipeline_tag: image-to-video tags: - world-model - robotics - teleoperation - video-generation - image-to-video - egocentric - diffusion base_model: Wan-AI/Wan2.2-TI2V-5B-Diffusers ---
## RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation

πŸ’« Project Page   |    πŸ€— Hugging Face    |    πŸ€– ModelScope  |  πŸš€ Video    |    πŸ“„ arXiv  

--- ## 🌟 Abstract We introduce **RynnWorld-Teleop**, a robot-centric generative world model that instantiates the paradigm of **digital teleoperation**β€”decoupling robot data collection from physical hardware constraints. By transforming an operator's real-time hand-pose stream into high-fidelity egocentric robotic videos from a single reference image, RynnWorld-Teleop enables the scaling of expert trajectories in a purely virtual environment. Our framework integrates depth-aware skeletal conditioning with a progressive human-to-robot training curriculum, allowing it to inherit rich manipulation priors from large-scale human datasets. To support interactive use, we distill the model into a causal, autoregressive student capable of real-time streaming. Policies trained exclusively on RynnWorld-Teleop synthetic data achieve effective zero-shot Sim2Real transfer, demonstrating its power as a high-fidelity data engine for scaling dexterous robotic learning. --- ## πŸ“° News * **[2026.07.07]** πŸ”₯πŸ”₯ Release our Technical Report !! * **[2026.07.07]** πŸ”₯πŸ”₯ Release our code and model checkpoints!! --- ## πŸ“¦ This Repository This repository hosts the **SFT (full fine-tune) checkpoint** of RynnWorld-Teleop. Given a first-frame image and a hand-pose / skeleton control video, the model generates a high-fidelity egocentric robotic video. ### Model Zoo | Model | HuggingFace | ModelScope | | :--------------- | :---------: | :--------: | | SFT | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnWorld-Teleop) | [Link](https://www.modelscope.cn/models/DAMO_Academy/RynnWorld-Teleop) | | Causal | [Link](https://huggingface.co/Alibaba-DAMO-Academy/RynnWorld-Teleop-Causal) | [Link](https://www.modelscope.cn/models/DAMO_Academy/RynnWorld-Teleop-Causal) | --- ## πŸš€ Quick Start Please refer to the [code repository](https://github.com/alibaba-damo-academy/RynnWorld-Teleop) for the full setup, training, and inference pipeline. ### πŸ”§ Environment Setup ```bash conda create -n "rynnworld-teleop" python=3.10 -y conda activate rynnworld-teleop pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu121 pip install -r requirements.txt ``` ### πŸ“– Download the Checkpoint Our model is developed on top of [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers). Download the base model and place it under `pretrained/`: ``` RynnWorld-Teleop/ └── pretrained/ └── Wan2.2-TI2V-5B-Diffusers/ β”œβ”€β”€ model_index.json β”œβ”€β”€ scheduler/ β”œβ”€β”€ transformer/ β”œβ”€β”€ vae/ └── ... ``` Then download our fine-tuned weights: ```bash mkdir -p pretrained/RynnWorld-Teleop huggingface-cli download Alibaba-DAMO-Academy/RynnWorld-Teleop --local-dir pretrained/RynnWorld-Teleop ``` --- ## 🎬 Inference Given a first-frame image and a control video (hand-pose / OpenPose mp4), the model generates the corresponding egocentric video. ```bash python inference_user.py \ --image \ --control_video \ --output results/my_demo \ --prompt "Describe the action in one sentence." \ --checkpoint \ --mode sft \ --control_type add \ --seeds "42,123,7" ``` **Required arguments** - `--image`: first-frame image (jpg/png), automatically resized to 832Γ—480 - `--control_video`: control video mp4 (hand-pose / OpenPose), sampled/interpolated to 81 frames - `--output`: output directory **Useful options** - `--mode sft|lora`: which checkpoint type to load (default: `sft`) - `--prompt`: optional natural-language description (encoded with the T5 text encoder) - `--text_embedding`: alternative pre-encoded prompt embedding `.safetensors` - `--seeds "42,123,7"`: generate multiple samples in one run - `--no_ema`: use raw weights instead of EMA - `--guidance_scale`: classifier-free guidance scale (default 1.0) - `--control_type add|concat|add-plus`: how the control signal is merged For **real-time streaming inference** with the distilled causal student, please see the [RynnWorld-Teleop-Causal](https://huggingface.co/Alibaba-DAMO-Academy/RynnWorld-Teleop-Causal) checkpoint. --- ## πŸ‹οΈ Training We train the teacher model in **three stages**: - **Stage 0 β€” Pretrain (egocentric human videos):** Full-parameter SFT on large-scale egocentric data, no control-video conditioning. Absorbs general manipulation priors. - **Stage 1 β€” Control-conditioned fine-tuning:** Adds a zero-initialized `control_patch_embedding` (Conv3d) and a learnable `control_scale` to inject hand-pose control video into the diffusion process. LoRA (lightweight) and Full-SFT (best quality) variants are provided. - **Distillation β†’ Causal student:** Distilled into a causal autoregressive model for real-time streaming. Full training scripts, configs, and data-preparation instructions are available in the [code repository](https://github.com/alibaba-damo-academy/RynnWorld-Teleop). --- ## πŸ“‘ Citation If you find this project useful, please cite: ```bibtex @article{rynnworld_teleop, title = {RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation}, author = {DAMO Academy, Alibaba Group}, year = {2026}, } ``` ## License Apache License 2.0 β€” see [LICENSE](https://github.com/alibaba-damo-academy/RynnWorld-Teleop/blob/main/LICENSE) for details.