ViTAL-X: Video-Text Alignment with Cross-Modal Temporal Edits
Sethuraman T, Savya Khosla, Onkar Kishor Susladkar, Aditi Tiwari, Seoung Wug Oh, Kushal Kafle, Joon-Young Lee, Derek Hoiem, Simon Jenni
Abstract
Video-text models adapted from image-text architectures (e.g., CLIP) frequently exhibit temporal blindness, the inability to perceive fundamental cues like order, direction, and motion dynamics. Standard datasets mask this limitation by enabling models to exploit static spatial shortcuts. To systematically evaluate this, we introduce XTE-Bench, a diagnostic probe revealing that even large-scale video-language models struggle with basic temporal reasoning, indicating that parameter scaling alone is insufficient to resolve this flaw. To address this, we propose Cross-Modal Temporal Edits (XTE), a self-supervised framework that injects precise temporal supervision. By performing synchronized video-text transformations, XTE generates hard temporal negatives without manual annotation. We instantiate this with ViTAL-X, a lightweight model that equips frozen image-text backbones with temporal awareness while preserving their foundational spatial knowledge. Across six temporal benchmarks, ViTAL-X achieves state-of-the-art performance. Utilizing only 0.4B parameters and 1M training clips, ViTAL-X outperforms 7B-parameter models and surpasses baselines trained on 600x more data. These results demonstrate that targeted, high-quality temporal alignment provides a highly efficient alternative to pure scaling.
Create a lesson
Related papers
SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models
Junchao Huang, Guian Fang, Shengju Qian et al.
Thinking in Pictures: A Systematic Benchmark for Reasoning-driven Image Generation
Yutong Liu, Nan Huang, Xu Cao et al.
PlantC2USeg: Cross-Scale Consistent Pre-Training for Few-Shot Unified Plant Point Cloud Segmentation
Yu Tian, Xintong Jiang, Jan Franklin Adamowski et al.
MuyBridge: Mobile Human Center-of-Mass Estimation from Monocular Video via Sparse Fusion
Aidan Bradshaw, Marco Giordano, David Rode et al.
RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation
Xiaolei Lang, Ze Kang, Zehao Huang et al.
Efficient All-in-One Weather Restoration using Spectral Harmonization
Paula Garrido-Mellado, Daniel Feijoo, Yuning Cui et al.