Logo image
Hallucination Reduction for LLM-Based Audio Understanding via Multimodal Direct Preference Optimization
Preprint   Open access

Hallucination Reduction for LLM-Based Audio Understanding via Multimodal Direct Preference Optimization

Bebe Cosgrove, Aaron Isidore Grace and Weiran Wang
arXiv
arXiv
10/02/2026
DOI: 10.48550/arxiv.2610.04004
url
https://doi.org/10.48550/arxiv.2610.04004View
Preprint (Author's original) This preprint has not been evaluated by subject experts through peer review. Preprints may undergo extensive changes and/or become peer-reviewed journal articles. Open Access

Abstract

Large Audio Language Models (LALMs) are prone to hallucinating and over-relying on text priors when simultaneously presented with audio and text inputs. To mitigate these hallucinations, we propose utilizing the multimodal Direct Preference Optimization (mDPO) objective, which forces the model to ground its generation in the acoustic input by contrasting intact and distorted audio counterparts. We extend this preference learning framework to the audio domain by applying a variety of acoustic perturbations. Evaluating the Qwen2-Audio backbone across the DCASE 2025 Challenge and AH Existence datasets, we demonstrate that extending mDPO to LALMs significantly enhances temporal reasoning in the complex DCASE dataset, and improves performance on basic existence verification in the AH benchmark. We identify temporal reversal, frequency masking, and random noise as the most effective perturbations. Ultimately, our approach achieves an absolute accuracy improvement of 14.0% on the DCASE 2025 dataset and 27.4% on AH Existence.
Computer Science - Sound

Details

Metrics

1 Record Views
Logo image