Journal article
Continual knowledge infusion into pre-trained biomedical language models
BIOINFORMATICS, Vol.38(2), pp.494-502
01/15/2022
DOI: 10.1093/bioinformatics/btab671
PMID: 34554186
Abstract
Motivation: Biomedical language models produce meaningful concept representations that are useful for a variety of biomedical natural language processing (bioNLP) applications such as named entity recognition, relationship extraction and question answering. Recent research trends have shown that the contextualized language models (e.g. BioBERT, BioELMo) possess tremendous representational power and are able to achieve impressive accuracy gains. However, these models are still unable to learn high-quality representations for concepts with low context information (i.e. rare words). Infusing the complementary information from knowledge-bases (KBs) is likely to be helpful when the corpus-specific information is insufficient to learn robust representations. Moreover, as the biomedical domain contains numerous KBs, it is imperative to develop approaches that can integrate the KBs in a continual fashion.
Results: We propose a new representation learning approach that progressively fuses the semantic information from multiple KBs into the pretrained biomedical language models. Since most of the KBs in the biomedical domain are expressed as parent-child hierarchies, we choose to model the hierarchical KBs and propose a new knowledge modeling strategy that encodes their topological properties at a granular level. Moreover, the proposed continual learning technique efficiently updates the concepts representations to accommodate the new knowledge while preserving the memory efficiency of contextualized language models. Altogether, the proposed approach generates knowledge-powered embeddings with high fidelity and learning efficiency. Extensive experiments conducted on bioNLP tasks validate the efficacy of the proposed approach and demonstrates its capability in generating robust concept representations.
Details
- Title: Subtitle
- Continual knowledge infusion into pre-trained biomedical language models
- Creators
- Kishlay Jha - University of VirginiaAidong Zhang - University of Virginia
- Resource Type
- Journal article
- Publication Details
- BIOINFORMATICS, Vol.38(2), pp.494-502
- DOI
- 10.1093/bioinformatics/btab671
- PMID
- 34554186
- NLM abbreviation
- Bioinformatics
- ISSN
- 1367-4803
- eISSN
- 1460-2059
- Publisher
- Oxford Univ Press
- Number of pages
- 9
- Grant note
- IIS-2008208; 1934600; 1938167; 1955151 / US National Science Foundation; National Science Foundation (NSF)
- Language
- English
- Date published
- 01/15/2022
- Academic Unit
- Electrical and Computer Engineering
- Record Identifier
- 9984295024302771
Metrics
29 Record Views