AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control
Yiyang Du, Yuxuan Kuang, Qitao Zhao, Yixuan Wang, Zhanqiu Guo, Yi Xu, Xin Li, Wenbin He, Liu Ren, Chenyan Xiong
Abstract
Frontier multimodal language agents such as GPT-6 Astra have demonstrated strong capabilities in controlling robots directly via low-level numerical actions without high-level primitives. However, existing benchmarks report only episode-level success rates, which show neither how these agents succeed nor where they fail. In this paper, we present a systematic action-level analysis of frontier agents on direct robot control across 12 manipulation benchmarks. We introduce AgentRoboCritic, an intent-grounded critique framework in which the policy agent states a local intent alongside each action, and a critic evaluates the action's outcome and failure mode against this intent. Our action-level analysis demonstrates the promising capabilities of frontier agents on direct robot control, including closed-loop recovery from local execution failures, reliable semantic grounding across diverse scenes, and strong zero-shot manipulation without task-specific adaptation. It also identifies their current limitations, including task-relevant physical execution bottlenecks such as grasping and alignment, and uneven recovery across failure modes. Finally, we validate our critic on a 400-action human-annotated benchmark and release a dataset of 12,844 annotated actions alongside a fine-tuned open-source critic, Muse-Glimmer-30B, whose failure-mode accuracy rises from 32.58% to 74.24%, enabling low-cost action-level analysis.
Create a lesson
Related papers
A Tutorial on Complementarity-Based Contact Models for Robotics
Onur Beker, A. René Geist, Michael Posa et al.
PGX: Physically Grounded Execution for Long-Horizon Humanoid Loco-Manipulation
Kai Lu, Chenyang Ma, Ioannis Havoutis
PhyDEX: A Benchmark for Physical Reasoning in Vision-Language-Action Models via Dynamic Extrinsic Dexterity
Hao Vo, Taisei Hanyu, Yuki Ikebe et al.
ACE-Brain-0.7: Learning Execution-Grounded Reasoning for Embodied Agents
Yinhe Wang, Shijun Shi, Feng Pan et al.
JARVIS: An Agent That Designs, Builds, and Tests Circuit Boards on Its Own
Shibo Zhou, Tianqi Hong, Zheng Yan et al.
An Open-Source ROS 2 Framework for Shared Control in Surgical Teleoperation: Integrating the Haply Inverse3 with the da Vinci Research Kit
Balint Varga, Sebastian Fritz, Gergely Kiss et al.