Conference proceeding
QUALIFIER: Question-Guided Self-Attentive Multimodal Fusion Network for Audio Visual Scene-Aware Dialog
2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.2503-2511
01/2022
DOI: 10.1109/WACV51458.2022.00256
Abstract
Audio video scene-aware dialog (AVSD) is a new but more challenging visual question answering (VQA) task because of the higher complexity of feature extraction and fusion brought by the additional modalities. Although recent methods have achieved early success in improving feature extraction technique for AVSD, the technique of feature fusion still needs further investigation. In this paper, inspired by the success of self-attention mechanism and the importance of understanding questions for VQA answering, we propose a question-guided self-attentive multi-modal fusion network (QUALIFIER) to improve the AVSD practice in the stage of feature fusion and answer generation. Specifically, after extracting features and learning a comprehensive feature for each modality, we first use the designed self-attentive multi-modal fusion (SMF) module to aggregate each feature with the correlated information learned from others. Later, by prioritizing the question feature, we concatenate it with each fused feature to guide the generation of a natural language response to the question. As for experimental results, QUALIFIER shows better performance than other baseline methods in the large-scale AVSD dataset named DSTC7. Additionally, the human evaluation and ablation study results also demonstrate the effectiveness of our network architecture.
Details
- Title: Subtitle
- QUALIFIER: Question-Guided Self-Attentive Multimodal Fusion Network for Audio Visual Scene-Aware Dialog
- Creators
- Muchao Ye - Pennsylvania State UniversityQuanzeng You - MicrosoftFenglong Ma - Pennsylvania State University
- Resource Type
- Conference proceeding
- Publication Details
- 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.2503-2511
- Publisher
- IEEE
- DOI
- 10.1109/WACV51458.2022.00256
- ISSN
- 2472-6737
- eISSN
- 2642-9381
- Language
- English
- Date published
- 01/2022
- Academic Unit
- Computer Science
- Record Identifier
- 9984696567202771
Metrics
1 Record Views