In the dissertation, a monotone spline-based nonparametric estimation method is proposed for analyzing longitudinal data with mixture distributions. The innovative and efficient algorithm combining the concept of projected Newton-Raphson algorithm with linear mixed model estimation method is developed to obtain the nonparametric estimation of monotone B-spline functions. This algorithm provides an efficient and flexible approach for modeling longitudinal data monotonically. An iterative 'one-step-forward' algorithm based on the K-means clustering is then proposed to classify mixture distributions of longitudinal data. This algorithm is computationally efficient, especially for data with a large number of underlying distributions. To quantify the disparity of underlying distributions of longitudinal data, we also propose an index measure on the basis of the aggregated areas under the curve (AAUC), which makes no distributional assumptions and fits the theme of nonparametric analysis.
An extensive simulation study is conducted to assess the empirical performance of our method under different AAUC values, covariance structures, and sample sizes. Finally, we apply the new approach in the PREDICT-HD study, a multi-site observational study of Huntington Disease (HD), to explore and assess clinical markers in motor and cognitive domains for the purpose of distinguishing participants at risk of HD from healthy subjects.