Journal article
Evaluating Modes of Observations Using Heirarchical Signal Detection Rater Models
Multivariate behavioral research, Vol.58(4), pp.843-858
07/04/2023
DOI: 10.1080/00273171.2022.2138253
PMID: 36502400
Abstract
Scoring of performance can be based on different modes of observation, which can include live and videotaped observations. Although live observations have been the traditional format of measuring examinee performance in education and in healthcare, videotaped observations provide educators and testing agencies the promise of unbiased and standardized evaluations, offering practical solutions to limitations of real-time scoring. This study proposes a measurement model taking into account different modes of observation, using an extension of the hierarchical rater model based on signal detection theory (HRM-SDT). A hierarchical rater model is motivated by the nested structure of scores assigned by raters - scores assigned by raters become indicators of performance quality, which in turn become indicators of examinee ability. This study extends the hierarchical structure of the scoring process to include modes of observation, which serves as an intermediary level between the latent categorical indicator of performance quality and examinee ability, forming a three-level HRM-SDT. Analyses based on real-world data showed differences in the quality of scores from live observations and videotaped recordings. Compared to the traditional HRM-SDT, the overall model fit improved when including modes of observation. Simulations using different sample sizes and conditions provide implications for uses of this model.
Details
- Title: Subtitle
- Evaluating Modes of Observations Using Heirarchical Signal Detection Rater Models
- Creators
- Yoon Soo Park - Harvard University PressQiao Lin - University of Illinois at ChicagoKuan Xing - University of Tennessee Health Science Center
- Resource Type
- Journal article
- Publication Details
- Multivariate behavioral research, Vol.58(4), pp.843-858
- Publisher
- Taylor & Francis
- DOI
- 10.1080/00273171.2022.2138253
- PMID
- 36502400
- ISSN
- 0027-3171
- eISSN
- 1532-7906
- Number of pages
- 16
- Language
- English
- Date published
- 07/04/2023
- Academic Unit
- Family and Community Medicine; Office of Consultation and Research in Medical Education
- Record Identifier
- 9984658249802771
Metrics
2 Record Views