Differential Weighting for Subcomponent Measures of Integrated Clinical Encounter Scores Based on the USMLE Step 2 CS Examination: Effects on Composite Score Reliability and Pass-Fail Decisions

Yoon Soo Park; Matthew Lineberry; Abbas Hyderi; Georges Bordage; Kuan Xing; Rachel Yudkowsky

doi:10.1097/ACM.0000000000001359

Back

Differential Weighting for Subcomponent Measures of Integrated Clinical Encounter Scores Based on the USMLE Step 2 CS Examination: Effects on Composite Score Reliability and Pass-Fail Decisions

Journal article

Open access

Peer reviewed

Differential Weighting for Subcomponent Measures of Integrated Clinical Encounter Scores Based on the USMLE Step 2 CS Examination: Effects on Composite Score Reliability and Pass-Fail Decisions

Yoon Soo Park, Matthew Lineberry, Abbas Hyderi, Georges Bordage, Kuan Xing and Rachel Yudkowsky

Academic medicine, Vol.91(11), pp.S24-S30

11/01/2016

DOI: 10.1097/ACM.0000000000001359

PMID: 27779506

Files and links (1)

url

https://doi.org/10.1097/ACM.0000000000001359View

Published (Version of record) Open Access

Abstract

Purpose Medical schools administer locally developed graduation competency examinations (GCEs) following the structure of the United States Medical Licensing Examination Step 2 Clinical Skills that combine standardized patient (SP)-based physical examination and the patient note (PN) to create integrated clinical encounter (ICE) scores. This study examines how different subcomponent scoring weights in a locally developed GCE affect composite score reliability and pass-fail decisions for ICE scores, contributing to internal structure and consequential validity evidence. Method Data from two M4 cohorts (2014: n = 177; 2015: n = 182) were used. The reliability of SP encounter (history taking and physical examination), PN, and communication and interpersonal skills scores were estimated with generalizability studies. Composite score reliability was estimated for varying weight combinations. Faculty were surveyed for preferred weights on the SP encounter and PN scores. Composite scores based on Kane's method were compared with weighted mean scores. Results Faculty suggested weighting PNs higher (60%-70%) than the SP encounter scores (30%-40%). Statistically, composite score reliability was maximized when PN scores were weighted at 40% to 50%. Composite score reliability of ICE scores increased by up to 0.20 points when SP-history taking (SP-Hx) scores were included; excluding SP-Hx only increased composite score reliability by 0.09 points. Classification accuracy for pass-fail decisions between composite and weighted mean scores was 0.77; misclassification was <5%. Conclusions Medical schools and certification agencies should consider implications of assigning weights with respect to composite score reliability and consequences on pass-fail decisions.

Education & Educational Research

Education, Scientific Disciplines

Health Care Sciences & Services

Life Sciences & Biomedicine

Science & Technology

Social Sciences

Details

Title: Subtitle: Differential Weighting for Subcomponent Measures of Integrated Clinical Encounter Scores Based on the USMLE Step 2 CS Examination: Effects on Composite Score Reliability and Pass-Fail Decisions
Creators: Yoon Soo Park - University of Illinois Chicago
Matthew Lineberry - University of Illinois Chicago
Abbas Hyderi - University of Illinois Chicago
Georges Bordage - University of Illinois Chicago
Kuan Xing - University of Illinois Chicago
Rachel Yudkowsky - University of Illinois Chicago
Resource Type: Journal article
Publication Details: Academic medicine, Vol.91(11), pp.S24-S30
DOI: 10.1097/ACM.0000000000001359
PMID: 27779506
NLM abbreviation: Acad Med
ISSN: 1040-2446
eISSN: 1938-808X
Publisher: Lippincott Williams & Wilkins
Number of pages: 7
Language: English
Date published: 11/01/2016
Academic Unit: Family and Community Medicine; Office of Consultation and Research in Medical Education
Record Identifier: 9984658251302771

Metrics

11 Record Views

16 Times Cited - Web of Science