Book chapter
Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions
Integration of Constraint Programming, Artificial Intelligence, and Operations Research, pp.200-218
Lecture Notes in Computer Science, Springer Nature Switzerland
05/23/2023
DOI: 10.1007/978-3-031-33271-5_14
Abstract
One surprising trait of neural networks is the extent to which their connections can be pruned with little to no effect on accuracy. But when we cross a critical level of parameter sparsity, pruning any further leads to a sudden drop in accuracy. This drop plausibly reflects a loss in model complexity, which we aim to avoid. In this work, we explore how sparsity also affects the geometry of the linear regions defined by a neural network, and consequently reduces the expected maximum number of linear regions based on the architecture. We observe that pruning affects accuracy similarly to how sparsity affects the number of linear regions and our proposed bound for the maximum number. Conversely, we find out that selecting the sparsity across layers to maximize our bound very often improves accuracy in comparison to pruning as much with the same sparsity in all layers, thereby providing us guidance on where to prune.
Details
- Title: Subtitle
- Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions
- Creators
- Junyang Cai - Bucknell UniversityKhai-Nguyen Nguyen - Bucknell UniversityNishant Shrestha - Bucknell UniversityAidan Good - Bucknell UniversityRuisen Tu - Bucknell UniversityXin Yu - University of UtahShandian Zhe - University of UtahThiago Serra - Bucknell University
- Contributors
- Andre A. Cire (Editor)
- Resource Type
- Book chapter
- Publication Details
- Integration of Constraint Programming, Artificial Intelligence, and Operations Research, pp.200-218
- Publisher
- Springer Nature Switzerland; Cham
- Series
- Lecture Notes in Computer Science
- DOI
- 10.1007/978-3-031-33271-5_14
- eISSN
- 1611-3349
- ISSN
- 0302-9743
- Language
- English
- Date published
- 05/23/2023
- Academic Unit
- Business Analytics
- Record Identifier
- 9984696757802771
Metrics
1 Record Views