Journal article
Ordered quantile normalization: a semiparametric transformation built for the cross-validation era
Journal of applied statistics, Vol.47(13-15), pp.2312-2327
11/17/2020
DOI: 10.1080/02664763.2019.1630372
PMCID: PMC9042069
PMID: 35707424
Abstract
Normalization transformations have recently experienced a resurgence in popularity in the era of machine learning, particularly in data preprocessing. However, the classical methods that can be adapted to cross-validation are not always effective. We introduce Ordered Quantile (ORQ) normalization, a one-to-one transformation that is designed to consistently and effectively transform a vector of arbitrary distribution into a vector that follows a normal (Gaussian) distribution. In the absence of ties, ORQ normalization is guaranteed to produce normally distributed transformed data. Once trained, an ORQ transformation can be readily and effectively applied to new data. We compare the effectiveness of the ORQ technique with other popular normalization methods in a simulation study where the true data generating distributions are known. We find that ORQ normalization is the only method that works consistently and effectively, regardless of the underlying distribution. We also explore the use of repeated cross-validation to identify the best normalizing transformation when the true underlying distribution is unknown. We apply our technique and other normalization methods via the
bestNormalize
R package on a car pricing data set. We built
bestNormalize
to evaluate the normalization efficacy of many candidate transformations; the package is freely available via the Comprehensive R Archive Network.
Details
- Title: Subtitle
- Ordered quantile normalization: a semiparametric transformation built for the cross-validation era
- Creators
- Ryan A Peterson - Department of Biostatistics and Informatics, Colorado School of Public Health, University of Colorado Anschutz Medical CampusJoseph E Cavanaugh - Department of Biostatistics, University of Iowa College of Public Health
- Resource Type
- Journal article
- Publication Details
- Journal of applied statistics, Vol.47(13-15), pp.2312-2327
- DOI
- 10.1080/02664763.2019.1630372
- PMID
- 35707424
- PMCID
- PMC9042069
- NLM abbreviation
- J Appl Stat
- ISSN
- 0266-4763
- eISSN
- 1360-0532
- Publisher
- Taylor & Francis
- Language
- English
- Date published
- 11/17/2020
- Academic Unit
- Statistics and Actuarial Science; Biostatistics; Injury Prevention Research Center
- Record Identifier
- 9984214953802771
Metrics
22 Record Views