Thompson sampling with smoothing splines for hyperparameter optimization: a tuning algorithm that doesn't need tuning

Machine learning methods are increasing in popularity for biomedical research. The task of hyperparameter optimization - informally referred to as “tuning” - is often the most laborious and time-consuming part of training machine learning models from the biostatistician’s perspective, but is critical for ensuring appropriate performance. The problem of finding optimal hyperparameters rarely admits a tractable formula, so computationally demanding methods like cross-validation and split-sample approaches are usually used instead.

In this dissertation, we propose a novel algorithm for tuning, in which a cubic smoothing spline regression model is used to relate hyperparameters to simple univariate transformations of validated performance, and Thompson sampling is used to decide which grid points to evaluate next. The theoretical properties of the procedure are discussed, and in empirical analyses, its performance is observed to compare favorably to random grid search and other state-of-the-art procedures. Additionally, it is shown how the method can be used asynchronously in parallel, and a stopping rule is devised. Our results demonstrate the potential of Thompson sampling with smoothing splines as a useful tool for tuning, and importantly, does not itself require tuning.

All software used for producing the results and methods in this dissertation, as well as data sets used, results, and an implementation of the proposed method as an R package, are provided online at https://github.com/matthewlouisdavisBioStat/TSSSr.