Journal article
Fast inference for quantile regression with tens of millions of observations
Journal of econometrics, Vol.249, p.105673
05/2025
DOI: 10.1016/j.jeconom.2024.105673
Abstract
Big data analytics has opened new avenues in economic research, but the challenge of analyzing datasets with tens of millions of observations is substantial. Conventional econometric methods based on extreme estimators require large amounts of computing resources and memory, which are often not readily available. In this paper, we focus on linear quantile regression applied to “ultra-large” datasets, such as U.S. decennial censuses. A fast inference framework is presented, utilizing stochastic subgradient descent (S-subGD) updates. The inference procedure handles cross-sectional data sequentially: (i) updating the parameter estimate with each incoming “new observation”, (ii) aggregating it as a Polyak–Ruppert average, and (iii) computing a pivotal statistic for inference using only a solution path. The methodology draws from time-series regression to create an asymptotically pivotal statistic through random scaling. Our proposed test statistic is calculated in a fully online fashion and critical values are calculated without resampling. We conduct extensive numerical studies to showcase the computational merits of our proposed inference. For inference problems as large as (n,d)∼(107,103), where n is the sample size and d is the number of regressors, our method generates new insights, surpassing current inference methods in computation. Our method specifically reveals trends in the gender gap in the U.S. college wage premium using millions of observations, while controlling over 103 covariates to mitigate confounding effects.
Details
- Title: Subtitle
- Fast inference for quantile regression with tens of millions of observations
- Creators
- Sokbae Lee - Columbia UniversityYuan Liao - Rutgers, The State University of New JerseyMyung Hwan Seo - Seoul National UniversityYoungki Shin - McMaster University
- Resource Type
- Journal article
- Publication Details
- Journal of econometrics, Vol.249, p.105673
- DOI
- 10.1016/j.jeconom.2024.105673
- ISSN
- 0304-4076
- eISSN
- 1872-6895
- Publisher
- Elsevier B.V
- Grant note
- Ministry of Education of the Republic of Korea (http://dx.doi.org/10.13039/501100002701) Center for Distributive Justice at the Institute of Economic Research, Seoul National University, Republic of Korea NRF-2018S1A5A2A01033487 / National Research Foundation of Korea (http://dx.doi.org/10.13039/501100003725)
- Language
- English
- Date published
- 05/2025
- Academic Unit
- Economics
- Record Identifier
- 9984936816702771
Metrics
1 Record Views