Logo image
Back
Separation-like irregularity and sample size optimism in high-discrimination logistic prediction models
Journal article   Open access   Peer reviewed

Separation-like irregularity and sample size optimism in high-discrimination logistic prediction models

Ye Liang, Louis Shuo Wang, Jiguang Yu and Xin Zan
PloS one, Vol.21(8), e0342286
08/24/2026
DOI: 10.1371/journal.pone.0342286
PMID: 42636249
url
https://doi.org/10.1371/journal.pone.0342286View
Published (Version of record) Open Access

Abstract

Closed-form minimum sample size criteria for developing logistic prediction models, such as the Riley framework implemented in pmsampsize, are widely used but may become optimistic when anticipated discrimination is high. We conducted a Monte Carlo simulation study to compare the formula-based recommended development sample size, nRiley, with an empirical required sample size, nreq, defined by out-of-sample calibration-slope stability under repeated development sampling. Scenarios fixed the candidate parameter dimension at p = 10 and crossed predictor distribution (normal, standardized skewed continuous, binary), signal density (dense versus sparse), prevalence (ϕ∈{0.05,0.10,0.20}), and target discrimination (AUCtarget∈{0.70,0.75,0.80,0.85,0.90}), with intercept and signal strength calibrated to match targets. We defined nreq as the smallest n such that 𝔼(bn)≥0.90 and Pr(bn<0.80)≤0.20, where bn is the truth-based logit-scale calibration slope evaluated on a large fixed validation covariate set of size Nval=50,000. At moderate discrimination, nRiley approximated nreq, but as discrimination increased the formula increasingly underestimated the sample size required for calibration stability, with large deficits at AUCtarget=0.90. Separation-like behavior (extreme fitted risks and linear predictors) at n=nRiley became common in high-discrimination settings despite nominal convergence, providing a plausible mechanism for formula optimism. These findings support augmenting formula-based planning with targeted simulation stress tests and instability diagnostics when high discrimination is anticipated.
 Computer Simulation Humans Logistic Models Monte Carlo Method Prediction Algorithms Sample Size

Details

Metrics

1 Record Views
Logo image