Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training

Abstract

Learning efficient and expressive visual representation has long been the pursuit of computer vision research. While Vision Transformers (ViTs) gradually replace traditional Convolutional Neural Networks (CNNs) as more scalable vision learners, their applications are plagued by the quadratic complexity of the self-attention mechanism. To address the challenge, we introduce a new linear-time sequence modeling method Test-Time Training (TTT) into vision and propose Vision-TTT, which treats visual sequences as datasets and compresses the visual token sequences in a novel self-supervised learning manner. By incorporating the dual-dataset strategy and Conv2d-based dataset preprocessing, Vision-TTT effectively extends vanilla TTT to model 2D visual correlations with global receptive fields. Extensive experiments show that Vittt-T/S/B achieve 77.7\%,81.8\%,82.7\% Top-1 accuracy on ImageNet classification and also greatly outperform their counterparts on downstream tasks. At 1280×1280 resolution, Vittt-T reduces FLOPs by 79.4\% and runs 4.72× faster with 88.9\% less memory than DeiT-T. These results demonstrate the expressiveness and efficiency of Vision-TTT as a strong candidate for the next-generation generic visual backbone.