Preprint
Evaluating WAIC and PSIS-LOO for Bayesian Diagnostic Classification Model Selection
ArXiv.org
Cornell University
10/03/2024
DOI: 10.48550/arxiv.2410.02931
Abstract
Bayesian diagnostic classification models (Bayesian DCMs) are effective for diagnosing students' skills. Research on the evaluation of relative model fit indices for DCMs using Bayesian estimation, however, is deficient. This study introduces the performance of Bayesian relative model fit indices, the widely applicable information criterion (WAIC) and leave-one-out cross-validation using Pareto-smoothed importance sampling (PSIS-LOO), in comparison to simpler and more widely used deviance information criterion (DIC). The simulation study evaluates the performance of WAIC and PSIS-LOO by detecting the true model with varying sample sizes, item qualities, and prior information levels. The results of the study indicate that WAIC and PSIS-LOO primarily favored the generating model; however, occasional inconsistencies were observed. This study recommends using WAIC and PSIS-LOO when the data is assumed to follow a simpler model and the models are estimated under uninformative priors, and DIC when the data is assumed to follow a more complex model.
Details
- Title: Subtitle
- Evaluating WAIC and PSIS-LOO for Bayesian Diagnostic Classification Model Selection
- Creators
- Ae Kyong JungJonathan Templin
- Resource Type
- Preprint
- Publication Details
- ArXiv.org
- DOI
- 10.48550/arxiv.2410.02931
- ISSN
- 2331-8422
- Publisher
- Cornell University; Ithaca, New York
- Language
- English
- Date posted
- 10/03/2024
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9984722575902771
Metrics
10 Record Views