Semantic Slots for Video Object-Centric Learning

Abstract

Video Object-Centric Learning (OCL) has traditionally focused on refining the encoder architecture to ensure temporal consistency. In this paper, we argue that the primary bottleneck lies in the decoder. We show that traditional decoders force slots to be spatially anchored, hindering their ability to adapt to motion. We propose SemanticSlots, which uses a Transformer-based decoder that leverages image context, relieving slots from encoding boundary precision and spatial location. This allows slots to function as semantic queries that are inherently object position invariant, retrieving matching features rather than memorizing coordinates. More importantly, this property allows slots computed from a single frame to decompose subsequent video frames, eliminating the need for complex temporal predictors or auxiliary temporal losses. Results on YouTube-VIS show that SemanticSlots improves upon VideoSAUR by 31 points in mBO and outperforms current state-of-the-art methods by 21 points, achieving 86.6% ARI and 62.8% mBO.