Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth
Peiyun Hu, Marko Mihajlovic, Lyndon Duong, Lahav Lipson, Wei Dong, Amaël Delaunoy, Stephan R. Richter, Vladlen Koltun
Abstract
We present Depth Pro 2, an approach to estimating dense metric depth for images and video at full resolution. By analyzing prior methods for monocular depth estimation, we find that they commonly trade off temporal consistency and boundary sharpness: multi-frame methods favor consistency, single-frame methods favor sharpness. To overcome this trade-off, we train a single, scalable network that processes both images and video and predicts depth maps at native resolution. Depth Pro 2 produces depth maps that are metrically accurate, sharp, and temporally consistent all at once, outperforming prior work in all of these dimensions. We release a family of models that define a new Pareto frontier in terms of metric accuracy, sharpness, temporal consistency, and runtime. We release code and weights at https://apple-aiml-research.github.io/ml-depth-pro-2/
Create a lesson
Related papers
Posterior Matching Distillation
Hidir Yesiltepe, Julius Berner, Arash Vahdat et al.
RL Post-Training in Diffusion Transformers Is Activation Steering
Tuna Han Salih Meral, Kaan Oktay, Alec Helbling et al.
One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning
Yuxin Jiang, Wenqiang Sun, Jiaqi Li et al.
ScaleWorld: High-Resolution World Generation through Shared Prediction and Refinement
Yansong Qu, Wesker Zheng, Quanjian Song et al.
Coda-DaVinci: Reimagining Video Generation with Code2Video
Yiyan Ji, Qixun Wang, Yang Shi et al.
Multi-institutional grading of Barrett's esophagus dysplasia without stain normalization using a pathology foundation model and multiple instance learning
Amirali Khosravi, Jaidip M. Jagtap, D. Chamil Codipilly et al.