Dissertation
Constrained optimization techniques for machine learning under error bound conditions
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Summer 2024
DOI: 10.25820/etd.007781
Abstract
The recent studies and innovations in the topic of machine learning (ML) demonstrates the capabilities of ML to analyze data, make predictions, and so on. To facilitate decisions, models in ML are often trained by solving a data-driven optimization problem. Moreover, motivated by various restrictions from applications in ML, we impose some conditions during the training of the model and conceptualize them as constraints in the optimization problem. This thesis seeks to strengthen the interaction between optimization and machine learning by studying first-order methods (FOMs) for optimization problems subject to various data-driven constraints.
In this thesis, we focus on analyzing and improving the oracle complexity and statistical property of FOMs, with an emphasis on the error bound conditions. In Chapter 2, we propose a new framework of federated learning (FL), which is an emerging technique in ML for training a model using data distributed over a network of nodes without sharing data between nodes. Our framework introduces personalized weights on other nodes to identify nodes whose data has high similarity and formulates this personalization as a bilevel optimization problem. We study the generalization performance of the model under the personalization and explore the error bound conditions under which our model benefits from this approach and outperforms models learned solely on the data of an individual node and through the non-personalized classical FL.
In Chapter 3, we propose a single-loop switching subgradient (SSG) method for non-smooth weakly convex functional constrained optimization problems. Most existing FOMs for this class of problems involve a double-loop implementation, which leads to high tuning effort and may require knowing some problem-dependent parameters that are hard to estimate. Our single-loop SSG method achieves the same complexity as the state-of-the-art double-loop approaches by utilizing local error bound conditions and is much easier to implement. The numerical results of applications to fairness-aware ML problems demonstrate our theoretical findings.
In Chapter 4, we study the smoothed proximal augmented Lagrangian method (SP-ALM) for smooth non-convex functional constrained optimization problems. We reduce the best-known complexity of this type of problems by considering linear error bound conditions. The numerical results of applications to fairness-aware ML problems demonstrate our theoretical findings.
Details
- Title: Subtitle
- Constrained optimization techniques for machine learning under error bound conditions
- Creators
- Yankun Huang
- Contributors
- Qihang Lin (Advisor)Samuel A Burer (Committee Member)Mingrui Liu (Committee Member)W. Nick Street (Committee Member)Renbo Zhao (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Business Administration (Business Analytics)
- Date degree season
- Summer 2024
- Publisher
- University of Iowa
- DOI
- 10.25820/etd.007781
- Number of pages
- x, 176 pages
- Copyright
- Copyright 2024 Yankun Huang
- Language
- English
- Date submitted
- 06/12/2024
- Description illustrations
- illustrations (some color)
- Description bibliographic
- Includes bibliographical references (pages 167-176).
- Public Abstract (ETD)
The recent studies and innovations in the topic of machine learning demonstrates the capabilities of machine learning to analyze data, make predictions, and so on. To facilitate decisions, models in machine learning are often trained by solving a data-driven optimization problem. Moreover, motivated by various restrictions from applications in machine learning, we impose some conditions during the training of the model and conceptualize them as constraints in the optimization problem. This thesis seeks to strengthen the interaction between optimization and machine learning by studying first-order methods for optimization problems subject to various data-driven constraints. The main contributions of this thesis are summarized as: personalization of models, single-loop implementation, and complexity reduction by smoothness and error bound conditions.
- Academic Unit
- Bus Admin Graduate Programs
- Record Identifier
- 9984698054502771
Metrics
2 File views/ downloads
11 Record Views