Journal article
Evaluating WAIC and PSIS-LOO for bayesian diagnostic classification model selection
Behaviormetrika
03/09/2026
DOI: 10.1007/s41237-025-00287-0
Abstract
Selecting an appropriate Diagnostic Classification Model (DCM) is critical for obtaining accurate feedback on student skills. For Bayesian DCMs, the standard Deviance Information Criterion (DIC) is limited as it summarizes the posterior distribution with a single point estimate, and it favors complex models. This study evaluates two alternatives that utilize the full posterior distribution: the Widely Applicable Information Criterion (WAIC) and Pareto-Smoothed Importance Sampling Leave-One-Out Cross-Validation (PSIS-LOO). A simulation study varied generating models, item quality, prior information level, sample size, and estimation models with Q-matrix specification to compare performance based on true model recovery and the classification accuracy of selected models. Results showed WAIC and PSIS-LOO were more often superior to DIC at identifying the correct model. Critically, models selected by WAIC and PSIS-LOO yielded more accurate student attribute and profile classifications, particularly with small samples and uninformative prior. This study provides strong evidence that WAIC and PSIS-LOO are preferable to DIC, leading to more reliable and valid diagnostic information.
Details
- Title: Subtitle
- Evaluating WAIC and PSIS-LOO for bayesian diagnostic classification model selection
- Creators
- Ae Kyong Jung - University of IowaJonathan Templin - University of Iowa
- Resource Type
- Journal article
- Publication Details
- Behaviormetrika
- DOI
- 10.1007/s41237-025-00287-0
- ISSN
- 0385-7417
- eISSN
- 1349-6964
- Publisher
- Springer Nature
- Language
- English
- Electronic publication date
- 03/09/2026
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9985144720602771
Metrics
1 Record Views