Conference proceeding
Stochastic Nonconvex Optimization with Large Minibatches
Proceedings of the 30th International Conference on Algorithmic Learning Theory, Vol.98, pp.857-882
Proceedings of Machine Learning Research
01/01/2019
Abstract
We study stochastic optimization of nonconvex loss functions, which are typical objectives for training neural networks. We propose stochastic approximation algorithms which optimize a series of regularized, nonlinearized losses on large minibatches of samples, using only first-order gradient information. Our algorithms provably converge to an approximate critical point of the expected objective with faster rates than minibatch stochastic gradient descent, and facilitate better parallelization by allowing larger minibatches.
Details
- Title: Subtitle
- Stochastic Nonconvex Optimization with Large Minibatches
- Creators
- Weiran Wang - Amazon Alexa, 101 Main St, Cambridge, MA 02142 USANathan Srebro - Kenwood
- Contributors
- A Garivier (Editor)S Kale (Editor)
- Resource Type
- Conference proceeding
- Publication Details
- Proceedings of the 30th International Conference on Algorithmic Learning Theory, Vol.98, pp.857-882
- Publisher
- Proceedings of Machine Learning Research
- Series
- Proceedings of Machine Learning Research
- ISSN
- 2640-3498
- eISSN
- 2640-3498
- Number of pages
- 26
- Language
- English
- Date published
- 01/01/2019
- Academic Unit
- Computer Science
- Record Identifier
- 9984696715802771
Metrics
1 Record Views