Temporal Contrastive Alignment of EEG and Naturalistic Video Only Beats Linear Decoding On In-Sample Stimuli
Yuanyuan Yao, Simon Geirnaert, Alexander Bertrand
Abstract
Neural tracking, the temporal alignment between a continuous stimulus and the elicited brain response, enables decoding of attention from electroencephalography (EEG), yet its application to naturalistic video remains largely unexplored and mainly relies on linear canonical correlation analysis (CCA). We investigate whether a temporal contrastive model, pairing pretrained VideoMAE features with a lightweight EEG encoder trained on a correlation-based InfoNCE objective, can strengthen EEG-video alignment. Benchmarked against CCA under three generalization regimes, the model outperforms CCA across subjects (leave-one-subject-out), matches it within the stimulus manifold (leave-video-segment-out), and falls behind CCA when extrapolating to entirely novel videos (leave-one-video-out). Because this advantage appears only for seen stimuli, we probe its basis: a sham control shows that a high accuracy does not certify genuine alignment, as a synthesized correspondence fit to the seen stimuli can reproduce it without genuine temporal alignment in the training pairs. This does not preclude practical use: on a downstream attention-decoding task with seen stimuli, the model remains better than CCA since the decoding only requires a strong statistical coupling, regardless of whether that coupling is physiologically meaningful. Using a group-averaged EEG reference, we further show that genuine, generalizable coupling is nonetheless learned, only buried in the noise of single-subject EEG. These results indicate that progress hinges on data scale, diversity, and single-subject denoising.
Create a lesson
Related papers
Decision Age: Information Freshness under Fusion Rules in Multi-Sensor Command and Control
Victor Monzon Baeza
Decision Fusion in Wireless Networks: A Tutorial and Survey of Classical and Learning-Based Methods
Mohamed Abdelkader, Arafat Al-Dweik, Sobia Jangsher et al.
Sensor-Free Shockable Rhythm Classification During CPR Using the Existing ECG Channel of Public-Access Automated External Defibrillators
Jaechan Lim, David Hicks, Matt Valentine et al.
Chance-Constrained Joint Beam Hopping Scheduling and Bandwidth Allocation for LEO Satellites
Olfa Ben Yahia, Jean-Francois Frigon, Brunilde Sanso et al.
Transition Dynamics Matter: Hidden Markov Scoring of Epileptogenic Channels in SEEG
Giulia Tacconi, Federico Mason, Lorenzo Ferri et al.
Reading Across Space, Time and Frequencies: Tensor Decomposition of SEEG Signals to Support Epileptogenic Zone Definition
Federico Mason, Matteo Neri, Maria Fratello et al.