Conference proceeding
A Unified Analysis of Stochastic Momentum Methods for Deep Learning
Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pp.2955-2961
07/2018
Abstract
Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochastic gradient (SG) method, and the stochastic momentum methods including two famous variants, i.e., the stochastic heavy-ball (SHB) method and the stochastic variant of Nesterov?s accelerated gradient (SNAG) method. We propose a framework that unifies the three variants. We then derive the convergence rates of the norm of gradient for the non-convex optimization problem, and analyze the generalization performance through the uniform stability approach. Particularly, the convergence analysis of the training objective exhibits that SHB and SNAG have no advantage over SG. However, the stability analysis shows that the momentum term can improve the stability of the learned model and hence improve the generalization performance. These theoretical insights verify the common wisdom and are also corroborated by our empirical analysis on deep learning.
Details
- Title: Subtitle
- A Unified Analysis of Stochastic Momentum Methods for Deep Learning
- Creators
- Yan YanTianbao YangZhe LiQihang LinYi Yang
- Resource Type
- Conference proceeding
- Publication Details
- Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pp.2955-2961
- Publisher
- International Joint Conferences on Artificial Intelligence Organization
- Language
- English
- Date published
- 07/2018
- Academic Unit
- Business Analytics; Computer Science
- Record Identifier
- 9984259413702771
Metrics
6 Record Views