GARD: Gradient-Guided Adversarial Representation Defense against Backdoors in Multimodal Contrastive Learning Models
Yuhong Wang, Yujun Cai, Lei Feng, Songze Li
Abstract
Multimodal contrastive learning models such as CLIP are vulnerable to backdoor attacks with diverse trigger structures, ranging from localized patches to globally distributed perturbations, with recent attacks increasingly optimizing these triggers in the representation space. Existing post-hoc defenses based on clean fine-tuning may leave backdoors active, while trigger-inversion methods depend on accurately estimating the attack target and trigger. We observe that backdoored encoders exhibit structure-dependent representation sensitivity: global perturbations preferentially expose distributed backdoors, whereas local patch perturbations are more effective for localized backdoors. Based on this observation, we propose Gradient-Guided Adversarial Representation Defense (GARD), a target- and trigger-agnostic post-hoc defense. GARD optimizes complementary global and local adversarial probes to maximally displace clean-image representations, and then realigns the perturbed representations with detached clean anchors while preserving image-text alignment. The resulting model requires no additional computation at inference time. Across six visual backdoor attacks and two CLIP visual backbones, GARD reduces the attack success rate below 1% in all 12 evaluated attack-backbone combinations while maintaining competitive clean accuracy. These results demonstrate the effectiveness of adversarial representation probing for repairing backdoored multimodal contrastive encoders.
Create a lesson
Related papers
SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research
Amine Lbath, Manan Suri, Aurelien Delaitre et al.
Text-Aware, Vision-Blind: How Multimodal Image Editing Systems Handle Unsafe Inputs
Ziqing Yang, Yixin Wu, Michael Backes et al.
GEOHunter: Mapping a Generative Engine Optimization Network in the Wild
Sean Tang, Peiran Wang, Ying Li et al.
Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)
Mengxiao Wang, Mahbub Alam, Kalyan Nakka et al.
Backtracking Agent Applications Security Incidents
Yicheng Cai, Taegyu Kim, Peng Liu
Visual Attribution Limits and Graded Provenance Auditing in Commercial Video-Generation APIs
Haocheng Mei, Yuchuan Tian, Yingte Shu et al.