Conference proceeding
VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 36 (NEURIPS 2023)
Advances in Neural Information Processing Systems
01/01/2023
Abstract
Vision-Language (VL) pre-trained models have shown their superiority on many multimodal tasks. However, the adversarial robustness of such models has not been fully explored. Existing approaches mainly focus on exploring the adversarial robustness under the white-box setting, which is unrealistic. In this paper, we aim to investigate a new yet practical task to craft image and text perturbations using pre-trained VL models to attack black-box fine-tuned models on different downstream tasks. Towards this end, we propose VLATTACK(2) to generate adversarial samples by fusing perturbations of images and texts from both single-modal and multimodal levels. At the single-modal level, we propose a new blockwise similarity attack (BSA) strategy to learn image perturbations for disrupting universal representations. Besides, we adopt an existing text attack strategy to generate text perturbations independent of the image-modal attack. At the multimodal level, we design a novel iterative cross-search attack (ICSA) method to update adversarial image-text pairs periodically, starting with the outputs from the single-modal level. We conduct extensive experiments to attack five widely-used VL pre-trained models for six tasks. Experimental results show that VLATTACK achieves the highest attack success rates on all tasks compared with state-of-the-art baselines, which reveals a blind spot in the deployment of pre-trained VL models.
Details
- Title: Subtitle
- VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models
- Creators
- Ziyi Yin - Pennsylvania State UniversityMuchao Ye - Pennsylvania State UniversityTianrong Zhang - Pennsylvania State UniversityTianyu Du - Zhejiang UniversityJinguo Zhu - Xi'an Jiaotong UniversityHan Liu - Dalian University of TechnologyJinghui Chen - Pennsylvania State UniversityTing Wang - Stony Brook UniversityFenglong Ma - Pennsylvania State University
- Contributors
- A Oh (Editor)T Neumann (Editor)A Globerson (Editor)K Saenko (Editor)M Hardt (Editor)S Levine (Editor)
- Resource Type
- Conference proceeding
- Publication Details
- ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 36 (NEURIPS 2023)
- Publisher
- Neural Information Processing Systems (Nips)
- Series
- Advances in Neural Information Processing Systems
- ISSN
- 1049-5258
- Number of pages
- 21
- Grant note
- 1951729; 1953813; 2119331; 2212323; 2238275 / National Science Foundation; National Science Foundation (NSF)
- Language
- English
- Date published
- 01/01/2023
- Academic Unit
- Computer Science
- Record Identifier
- 9984696826802771
Metrics
1 Record Views