Attribution-Guided Shared-Private Decoupling for Noise-Reduced Audio-Visual Representation Learning
Linge Wang, Yingying Chen, Bingke Zhu, Lu Zhou, Haixin Wang, Zhipiao Liu, Jinqiao Wang
Abstract
Audio-visual representation learning commonly aligns paired audio and visual signals with sample-level contrastive objectives. However, real-world audio-visual pairs usually contain heterogeneous patch tokens: only a subset carries reliable cross-modal shared semantics, while many others mainly encode modality-private residuals. Aggregating all tokens into a single global representation can therefore introduce semantic noise into cross-modal alignment. Motivated by the minimal-sufficiency principle, we propose an attribution-guided shared-private decoupling framework to mitigate this issue. Our key idea is to identify shared-cover subviews that preserve dominant cross-modal information while suppressing modality-private residuals, and use them to supervise decoupled shared and private representations. To identify such subviews efficiently, we draw on transformer attribution method and introduce Layer-Truncated Attribution (LTA), which provides a lightweight estimate of token-level relevance to cross-modal shared information. We further adopt a teacher-student framework, where the teacher provides attribution-induced supervision and the student internalizes shared-private decoupling into its representations, without introducing extra inference cost. Experiments on retrieval, classification, and sound-prompted semantic segmentation show consistent gains over strong baselines. Compared with baseline, our method improves the average zero-shot R@1 by 4.0 across AudioSet and VGGSound retrieval, and improves AS20K classification mAP by 3.1, demonstrating the effectiveness of fine grained attribution guidance for reducing semantic noise.reducing semantic noise. Code will be released at https://github.com/wanglg20/LTA
Create a lesson
Related papers
ZoionNET: Low-Power Animal Intrusion Detection Using an Acoustic Deep-Learning Classifier for a Cellular Remote-Sensing System
Prasanna Vijay, Adarsh Gupta, Vansh Panwar et al.
When Sounds Turn Heads: Recovering Event Timing That Audio LLMs Fail to Say
Jongbhin Woo, Jaewook Jeong, Joon Son Chung
From Attention to Semantics: Detecting and Mitigating Sound Event Hallucinations in Large Audio-Language Models
Tianyi Sun, Minghao Lv, Chunyang Yu et al.
The landscape of AI speech generation and deepfake detection
Simon King
Exploring Speaker-Aware Engagement Modeling for Children with Autism Spectrum Disorder
Kaleen Shrestha, Austin Joel Dympep, Young Suh Hong et al.
Harmonic Circuit: Chord Progression Lies in a Sparse Principal Information Flow in Symbolic Music Models
Ruiji Yu, Gus Xia