Online Bayesian logistic regression for streaming data

In many real-world situations, data keeps arriving continuously over time. For example, students submitting answers and completing activities on an online learning platform, with new responses streaming in constantly throughout the day, or sensors monitoring air quality around the clock. As new data comes in, we want to update our understanding quickly without having to reanalyze everything from scratch. Traditional methods struggle here because they require going back through all past data every time something new arrives, which gets slow and impractical as the data grows.

Our methods are specifically designed for situations where the outcome of interest has two possible categories. For example, whether a student will answer future questions correctly or not, or whether a fire was detected or not. We developed two new methods that solve the reprocessing problem by keeping only a small, efficient summary of past data and using it to update conclusions as each new batch arrives. This makes the process much faster and less memory-intensive, without sacrificing the quality of the results.

We tested our methods on simulated data, both in simple settings with few variables and more complex settings with many variables. We also test them on real datasets from education and fire detection. In all cases, our methods produced results just as accurate as the traditional approach, but in a fraction of the time.