EgoGeneralist: Unifying Egocentric Action-Video Tasks through Video-Native Modeling
Yanlin Jin, Bowen Yang, Cheng Zhang, Yifan Yang, Zhe Li, Jianfei Cai, Kai Zhu
Abstract
Modern video models possess strong visual priors, yet do not natively represent human action in metric 3D space. We present EgoGeneralist, which adapts pretrained video generators to unify action-conditioned video generation, pose estimation, and joint action-video generation for egocentric manipulation. Our role-adaptive action renderer provides video-native representations for multidirectional action-video learning, making metric motion usable as both a spatial control signal and a recoverable generation target. Cross-stream action-video coupling establishes close correspondence between metric motion and visual dynamics through lightweight adaptation while preserving the pretrained visual prior. Together, these designs establish a generalist framework spanning six action-video tasks. With only narrow-domain data, EgoGeneralist generalizes to diverse unseen scenes and interactions with effective action control and visual quality, bridging pretrained visual knowledge and metric 3D human motion.
Create a lesson
Related papers
Posterior Matching Distillation
Hidir Yesiltepe, Julius Berner, Arash Vahdat et al.
RL Post-Training in Diffusion Transformers Is Activation Steering
Tuna Han Salih Meral, Kaan Oktay, Alec Helbling et al.
One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning
Yuxin Jiang, Wenqiang Sun, Jiaqi Li et al.
ScaleWorld: High-Resolution World Generation through Shared Prediction and Refinement
Yansong Qu, Wesker Zheng, Quanjian Song et al.
Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth
Peiyun Hu, Marko Mihajlovic, Lyndon Duong et al.
Coda-DaVinci: Reimagining Video Generation with Code2Video
Yiyan Ji, Qixun Wang, Yang Shi et al.