Preprint
Massive End-to-end Models for Short Search Queries
arXiv (Cornell University)
09/22/2023
DOI: 10.48550/arxiv.2309.12963
Abstract
In this work, we investigate two popular end-to-end automatic speech recognition (ASR) models, namely Connectionist Temporal Classification (CTC) and RNN-Transducer (RNN-T), for offline recognition of voice search queries, with up to 2B model parameters. The encoders of our models use the neural architecture of Google's universal speech model (USM), with additional funnel pooling layers to significantly reduce the frame rate and speed up training and inference. We perform extensive studies on vocabulary size, time reduction strategy, and its generalization performance on long-form test sets. Despite the speculation that, as the model size increases, CTC can be as good as RNN-T which builds label dependency into the prediction, we observe that a 900M RNN-T clearly outperforms a 1.8B CTC and is more tolerant to severe time reduction, although the WER gap can be largely removed by LM shallow fusion.
Details
- Title: Subtitle
- Massive End-to-end Models for Short Search Queries
- Creators
- Weiran WangRohit PrabhavalkarDongseong HwangQiujia LiKhe Chai SimBo LiJames QinXingyu CaiAdam StookeZhong MengC J ZhengYanzhang HeTara SainathPedro Moreno Mengibar
- Resource Type
- Preprint
- Publication Details
- arXiv (Cornell University)
- DOI
- 10.48550/arxiv.2309.12963
- eISSN
- 2331-8422
- Language
- English
- Date posted
- 09/22/2023
- Academic Unit
- Computer Science
- Record Identifier
- 9984696798502771
Metrics
1 Record Views