Conference proceeding
Distributed Optimization of Deeply Nested Systems
Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, Vol.33, pp.10-19
JMLR Workshop and Conference Proceedings
01/01/2014
Abstract
Intelligent processing of complex signals such as images is often performed by a hierarchy of non-linear processing layers, such as a deep net or an object recognition cascade. Joint estimation of the parameters of all the layers is a difficult nonconvex optimization. We describe a general strategy to learn the parameters and, to some extent, the architecture of nested systems, which we call the method of auxiliary coordinates (MAC). This replaces the original problem involving a deeply nested function with a constrained problem involving a different function in an augmented space without nesting. The constrained problem may be solved with penalty-based methods using alternating optimization over the parameters and the auxiliary coordinates. MAC has provable convergence, is easy to implement reusing existing algorithms for single layers, can be parallelized trivially and massively, applies even when parameter derivatives are not available or not desirable, can perform some model selection on the fly, and is competitive with state-of-the-art nonlinear optimizers even in the serial computation setting, often providing reasonable models within a few iterations.
Details
- Title: Subtitle
- Distributed Optimization of Deeply Nested Systems
- Creators
- Miguel A. Carreira-Perpinan - Univ Calif Merced, Sch Engn, Elect Engn & Comp Sci, Merced, CA 95343 USAWeiran Wang - Univ Calif Merced, Sch Engn, Elect Engn & Comp Sci, Merced, CA 95343 USA
- Contributors
- S Kaski (Editor)J Corander (Editor)
- Resource Type
- Conference proceeding
- Publication Details
- Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, Vol.33, pp.10-19
- Publisher
- Proceedings of Machine Learning Research
- Series
- JMLR Workshop and Conference Proceedings
- ISSN
- 2640-3498
- Number of pages
- 10
- Grant note
- IIS-0754089 / NSF CAREER award; National Science Foundation (NSF); NSF - Office of the Director (OD)
- Language
- English
- Date published
- 01/01/2014
- Academic Unit
- Computer Science
- Record Identifier
- 9984696565502771
Metrics
1 Record Views