Conference proceeding
AdaRA: Adaptive Rank Allocation of Residual Adapters for Speech Foundation Model
INTERSPEECH 2024, pp.2375-2379
Interspeech
01/01/2024
DOI: 10.21437/Interspeech.2024-45
Abstract
A recent paradigm shift in artificial intelligence has witnessed the emergence of foundation models. These foundation models, possessing billions of parameters and trained on extensive datasets, are anticipated to demonstrate superior generalization across diverse downstream tasks. Residual Adapters represent a broadly employed methodology for efficient adaptation, achieved by updating a limited set of additive parameters while maintaining a fixed bottleneck dimension. However, when the parameter budget is constrained, allocating additive parameters uniformly across layers proves sub-optimal. In this paper, we propose a novel adaptive efficient adaptation method that automatically determines the optimal number of bottleneck dimensions for Residual Adapters at different layers. Experimental results confirm that the proposed method effectively learns an optimal additive parameter allocation, surpassing the performance of comparable methods in speech recognition domain adaptation.
Details
- Title: Subtitle
- AdaRA: Adaptive Rank Allocation of Residual Adapters for Speech Foundation Model
- Creators
- Zhouyuan Huo - GoogleDongseong Hwang - GoogleGan Song - GoogleKhe Chai Sim - GoogleWeiran Wang - Google
- Resource Type
- Conference proceeding
- Publication Details
- INTERSPEECH 2024, pp.2375-2379
- Publisher
- Isca-Int Speech Communication Assoc
- Series
- Interspeech
- DOI
- 10.21437/Interspeech.2024-45
- ISSN
- 2308-457X
- Number of pages
- 5
- Language
- English
- Date published
- 01/01/2024
- Academic Unit
- Computer Science
- Record Identifier
- 9984798226802771
Metrics
2 Record Views