From Festival Video to the Stage: A Sim-to-Real Ladder for Long-Horizon Expressive Whole-Body Dance on a Commercial Humanoid
In the authors' words
We present a pipeline for transferring Awa Odori, a traditional Japanese festival dance, from monocular video to a commercial Unitree G1 EDU humanoid. The pipeline combines human mesh recovery, kinematic retargeting, reinforcement learning for motion tracking in simulation, and onboard deployment via ONNX. Training ran on a single RTX 5070 GPU. An eleven-version deployment ladder evaluated on hardware examines three interventions. We detect and repair silent corruption of deployment artifacts, model state-estimator drift during training, and edit the reference motion to support stable execution and a controlled ending. The final policies achieved three consecutive rope-free completions of a 43-second routine, a 46-second performance with automatic handoff, and a continuous 9.7-minute rope-free performance composed from the captured motion. In a live handover, a single remote request let the robot start from and return to the manufacturer's walking controller on stage, with a measured torque-free window of 45-140 ms. We report the training dynamics and hardware outcomes and discuss failure boundaries involving compliant flooring and occlusion in monocular capture.
In plain words AI summary
The authors built a system to teach a commercial humanoid robot a traditional Japanese festival dance from ordinary video. Their pipeline uses computer vision, motion retargeting, and reinforcement learning in simulation before deploying to the real robot. They tested eleven versions on hardware, fixing problems like corrupted files, drifting sensors, and unstable motions. The final robot performed the 43-second dance three times without support ropes, and once for 9.7 minutes continuously. In a live demo, a remote command let the robot switch between dancing and normal walking, with 45-140 milliseconds of torque-free transition.
Written by Kimi K2.5 using only the abstract above. Jev checked that every sentence is supported by the abstract sentences it cites, and removed any that were not. Point at a sentence to highlight its source.
Appeared: Tuesday, November 10. arXiv. Preprint, not yet peer-reviewed.
Authors' comment: 19 pages, 12 figures. Four supplementary videos are linked in the manuscript