ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control
Xu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao, Jingkuan Song
Abstract
Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.
Create a lesson
Related papers
AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control
Yiyang Du, Yuxuan Kuang, Qitao Zhao et al.
A Tutorial on Complementarity-Based Contact Models for Robotics
Onur Beker, A. René Geist, Michael Posa et al.
PGX: Physically Grounded Execution for Long-Horizon Humanoid Loco-Manipulation
Kai Lu, Chenyang Ma, Ioannis Havoutis
PhyDEX: A Benchmark for Physical Reasoning in Vision-Language-Action Models via Dynamic Extrinsic Dexterity
Hao Vo, Taisei Hanyu, Yuki Ikebe et al.
ACE-Brain-0.7: Learning Execution-Grounded Reasoning for Embodied Agents
Yinhe Wang, Shijun Shi, Feng Pan et al.
JARVIS: An Agent That Designs, Builds, and Tests Circuit Boards on Its Own
Shibo Zhou, Tianqi Hong, Zheng Yan et al.