Journal article
Generalizability of Competency Assessment Scores Across and Within Clerkships: How Students, Assessors, and Clerkships Matter
Academic medicine, Vol.93(8), pp.1212-1217
08/01/2018
DOI: 10.1097/ACM.0000000000002262
PMID: 29697428
Abstract
Purpose
Many factors influence the reliable assessment of medical students' competencies in the clerkships. The purpose of this study was to determine how many clerkship competency assessment scores were necessary to achieve an acceptable threshold of reliability.
Method
Clerkship student assessment data were collected during the 2015-2016 academic year as part of the medical school assessment program at the University of Michigan Medical School. Faculty and residents assigned competency assessment scores for third-year core clerkship students. Generalizability (G) and decision (D) studies were conducted using balanced, stratified, and random samples to examine the extent to which overall assessment scores could reliably differentiate between students' competency levels both within and across clerkships.
Results
In the across-clerkship model, the residual error accounted for the largest proportion of variance (75%), whereas the variance attributed to the student and student-clerkship effects was much smaller (7% and 10.1%, respectively). D studies indicated that generalizability estimates for eight assessors within a clerkship varied across clerkships (G coefficients range = 0.000-0.795). Within clerkships, the number of assessors needed for optimal reliability varied from 4 to 17.
Conclusions
Minimal reliability was found in competency assessment scores for half of clerkships. The variability in reliability estimates across clerkships may be attributable to differences in scoring processes and assessor training. Other medical schools face similar variation in assessments of clerkship students; therefore, the authors hope this study will serve as a model for other institutions that wish to examine the reliability of their clerkship assessment scores.
Details
- Title: Subtitle
- Generalizability of Competency Assessment Scores Across and Within Clerkships: How Students, Assessors, and Clerkships Matter
- Creators
- Nikki L. Bibler Zaidi - University of Michigan–Ann ArborClarence D. Kreiter - Univ Iowa, Carver Coll Med, Off Consultat & Res Med Educ, Iowa City, IA USAPeris R. Castaneda - University of Michigan–Ann ArborJocelyn H. Schiller - University of Michigan–Ann ArborJun Yang - University of Michigan–Ann ArborCyril M. Grum - University of Michigan–Ann ArborMaya M. Hammoud - University of Michigan–Ann ArborLarry D. Gruppen - University of Michigan–Ann ArborSally A. Santen - Virginia Commonwealth University
- Resource Type
- Journal article
- Publication Details
- Academic medicine, Vol.93(8), pp.1212-1217
- Publisher
- Lippincott Williams & Wilkins
- DOI
- 10.1097/ACM.0000000000002262
- PMID
- 29697428
- ISSN
- 1040-2446
- eISSN
- 1938-808X
- Number of pages
- 6
- Language
- English
- Date published
- 08/01/2018
- Academic Unit
- Family and Community Medicine
- Record Identifier
- 9984297449302771
Metrics
12 Record Views