Journal article
A Proposal for Evaluating the Validity of Holistic-Based Admission Processes
Teaching and learning in medicine, Vol.25(1), pp.103-107
01/01/2013
DOI: 10.1080/10401334.2012.741548
PMID: 23330903
Abstract
Background. Admission decisions require that information about an applicant be combined using either holistic (human judges) or statistical (actuarial) methods. For optimizing a defined measureable outcome, there is a consistent body of research evidence demonstrating that statistical methods yield superior decisions compared to those generated by judges. It is possible, however, that the benefits of holistic decisions are reflected in unmeasured outcomes. If such benefits exist, they would necessarily appear as systematic variance in raters scores that deviate from statistically-based decisions. Purpose. To estimate this variance, we propose a design examining the interrater reliability of difference scores (i.e., the difference between observed committee rankings and rankings based on statistical approaches). Methods. Example calculations and G study models are presented to demonstrate how rater agreement on difference scores can be analyzed under various circumstances. High interrater reliability of difference scores would support but not prove the assertion that the holistic process adds useful information beyond that achieved by much less costly statistical approaches. Conversely, if the interrater reliability of difference scores is near zero, this would clearly demonstrate that committee judgments add random error to the decision process. Results. Evidence to conduct such studies already exists within most highly selective medical schools and graduate programs and the proposed validity research could be conducted on existing data. Conclusions. Such research evidence is critical for establishing the validity of widely used holistic admission approaches.
Details
- Title: Subtitle
- A Proposal for Evaluating the Validity of Holistic-Based Admission Processes
- Creators
- Clarence D. Kreiter - Roy J. and Lucille A. Carver College of Medicine
- Resource Type
- Journal article
- Publication Details
- Teaching and learning in medicine, Vol.25(1), pp.103-107
- DOI
- 10.1080/10401334.2012.741548
- PMID
- 23330903
- NLM abbreviation
- Teach Learn Med
- ISSN
- 1040-1334
- eISSN
- 1532-8015
- Publisher
- Taylor & Francis
- Number of pages
- 5
- Language
- English
- Date published
- 01/01/2013
- Academic Unit
- Family and Community Medicine
- Record Identifier
- 9984297338502771
Metrics
19 Record Views