Logo image
Back
Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generation
Conference proceeding   Open access

Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generation

Xingyi Yang, Muchao Ye, Quanzeng You and Fenglong Ma
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.5000-5009
08/2021
DOI: 10.18653/v1/2021.acl-long.387
url
https://doi.org/10.18653/v1/2021.acl-long.387View
Published (Version of record) Open Access

Abstract

Medical report generation is one of the most challenging tasks in medical image analysis. Although existing approaches have achieved promising results, they either require a predefined template database in order to retrieve sentences or ignore the hierarchical nature of medical report generation. To address these issues, we propose MedWriter that incorporates a novel hierarchical retrieval mechanism to automatically extract both report and sentencelevel templates for clinically accurate report generation. MedWriter first employs the Visual-Language Retrieval (VLR) module to retrieve the most relevant reports for the given images. To guarantee the logical coherence between sentences, the Language-Language Retrieval (LLR) module is introduced to retrieve relevant sentences based on the previous generated description. At last, a language decoder fuses image features and features from retrieved reports and sentences to generate meaningful medical reports. We verified the effectiveness of our model by automatic evaluation and human evaluation on two datasets, i.e., Open-I and MIMIC-CXR.
 Computer Science Social Sciences Technology Computer Science, Artificial Intelligence Computer Science, Interdisciplinary Applications Linguistics Science & Technology

Details

Metrics

13 Record Views
22 Times Cited - Web of Science
Logo image