Preprint
VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models
ArXiV.org
Cornell University
12/01/2024
DOI: 10.48550/arxiv.2412.01095
Abstract
The rapid advancement of vision-language models (VLMs) has established a new paradigm in video anomaly detection (VAD): leveraging VLMs to simultaneously detect anomalies and provide comprehendible explanations for the decisions. Existing work in this direction often assumes the complex reasoning required for VAD exceeds the capabilities of pretrained VLMs. Consequently, these approaches either incorporate specialized reasoning modules during inference or rely on instruction tuning datasets through additional training to adapt VLMs for VAD. However, such strategies often incur substantial computational costs or data annotation overhead. To address these challenges in explainable VAD, we introduce a verbalized learning framework named VERA that enables VLMs to perform VAD without model parameter modifications. Specifically, VERA automatically decomposes the complex reasoning required for VAD into reflections on simpler, more focused guiding questions capturing distinct abnormal patterns. It treats these reflective questions as learnable parameters and optimizes them through data-driven verbal interactions between learner and optimizer VLMs, using coarsely labeled training data. During inference, VERA embeds the learned questions into model prompts to guide VLMs in generating segment-level anomaly scores, which are then refined into frame-level scores via the fusion of scene and temporal contexts. Experimental results on challenging benchmarks demonstrate that the learned questions of VERA are highly adaptable, significantly improving both detection performance and explainability of VLMs for VAD.
Details
- Title: Subtitle
- VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models
- Creators
- Muchao Ye - University of IowaWeiyang LiuPan He - Auburn University
- Resource Type
- Preprint
- Publication Details
- ArXiV.org
- DOI
- 10.48550/arxiv.2412.01095
- ISSN
- 2331-8422
- Publisher
- Cornell University; Ithaca, New York
- Language
- English
- Date posted
- 12/01/2024
- Academic Unit
- Computer Science
- Record Identifier
- 9984756236502771
Metrics
21 Record Views