Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies
Chubin Zhang, Bo An
Abstract
Memory-augmented vision-language-action (VLA) policies are meant to remember what happened. Many of them, however, fill memory by time: they sample frames uniformly across the episode and stamp each frame with its time step. Because time passes even when nothing happens, such a memory may not tell waiting from working. We show that it does not. On the counting tasks of RoboMME, we make the robot briefly pause or fidget its wrist. Such a null event changes nothing the task depends on, yet a released memory VLA counts the interval as work done and stops early. We call this failure phantom progress. It cuts success from 47 to 16 of 50 on one task and from 44 to 10 on the other. When the same frames are withheld from memory, success returns, so the cause lies in memory rather than in the interruption itself. We trace the failure to a temporal shortcut: because demonstrations proceed at a steady pace, elapsed time predicts progress. Indeed, rescaling the time stamps alone makes the policy stop early when the clock runs fast and repeat when it runs slow. Surprisingly, removing the time encoding does not cure the failure, because the frames of a null event still occupy slots of the time-sampled memory. Across RoboMME's released designs, vulnerability follows how memory is indexed. Of the fixes the diagnosis suggests, training with injected pauses does not remove the failure and impairs counting, whereas indexing memory by events recovers half or more of the loss without retraining. A memory meant to remember what happened should count events rather than clock ticks.
Create a lesson
Related papers
AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control
Yiyang Du, Yuxuan Kuang, Qitao Zhao et al.
A Tutorial on Complementarity-Based Contact Models for Robotics
Onur Beker, A. René Geist, Michael Posa et al.
PGX: Physically Grounded Execution for Long-Horizon Humanoid Loco-Manipulation
Kai Lu, Chenyang Ma, Ioannis Havoutis
PhyDEX: A Benchmark for Physical Reasoning in Vision-Language-Action Models via Dynamic Extrinsic Dexterity
Hao Vo, Taisei Hanyu, Yuki Ikebe et al.
ACE-Brain-0.7: Learning Execution-Grounded Reasoning for Embodied Agents
Yinhe Wang, Shijun Shi, Feng Pan et al.
JARVIS: An Agent That Designs, Builds, and Tests Circuit Boards on Its Own
Shibo Zhou, Tianqi Hong, Zheng Yan et al.