New parallel algorithms for support vector machines and neural architecture search

Many of the most important questions being tackled by researchers today require large amounts of both data and computational resources. We are not limited in access to the resources needed to solve these problems but instead we are limited by time. These challenges range from the tuning of models, such as designing a neural network or tuning the hyper-parameters of a classical machine learning algorithm, to the sheer computational complexity of the problem, such as designing an artificial intelligence to play a board game such as chess or Go.

In many of these settings, researchers develop bespoke algorithms or system designs for their specific problem. This has certainly proven effective; however, designing more generic algorithms and systems that can be applied in a number of research settings can free researchers to focus on the problem at hand, rather than how they will scale their system to handle the computational demands of their research. This means progress where we need it–solving new and novel problems–rather than re-inventing the wheel with respect to algorithms and systems.

This thesis proposes several new parallel algorithms for Support Vector Machines (SVMs) and neural architecture search. Although these algorithms are developed in specific contexts, they are generic enough to be useful in a wide range of applications. The motivation for these new algorithms center around two ideas: speeding up the training and evaluation of models using multiple computers and designing systems such that computational resources can be added or removed from the system as they become available or are no longer needed. This is done with a focus on simplicity both in the algorithms and their implementations.

The effectiveness of this work is demonstrated on several fronts. In the first part of this thesis we focus on efficiently building and evaluating a large number of SVMs in parallel. The challenge is two fold: efficiently training a large number of SVMs and, once they are trained, efficiently evaluating them while inference requests flow into the system. The second part of this thesis tackles the problem of automating the design of neural networks. This is a difficult problem because of the computational demand of training and evaluating a large number of neural networks. Neural networks are particularly computationally intensive machine learning models that can take anywhere from minutes to days to train. The techniques we develop in this part of the thesis, while somewhat specialized to neural architecture search, are simple enough to understand and implement that they are easily transferred to other machine learning model types and problem domains.