Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions

Kathryn J. Smith; Brent N. Reed; Stephen Neely; Michelle Z. Farland; Stuart T. Haines; Jennifer D. Robinson

doi:10.1016/j.ajpe.2023.100129

Back

Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions

Journal article

Peer reviewed

Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions

Kathryn J. Smith, Brent N. Reed, Stephen Neely, Michelle Z. Farland, Stuart T. Haines and Jennifer D. Robinson

American journal of pharmaceutical education, Vol.87(11), pp.100129-100129

11/01/2023

DOI: 10.1016/j.ajpe.2023.100129

PMID: 37914464

Files and links (1)

url

https://doi.org/10.1016/j.ajpe.2023.100129View

Published (Version of record) Open Access

Abstract

The primary objective of this study was to examine the levels of agreement and reliability of a situational judgment test (SJT) using a diverse pool of pharmacy practice faculty as subject matter experts. Secondary aims included analyses to build support for test validity and fairness. An SJT containing 18 scenarios and 118 responses assessing empathy, integrity, and teamwork was developed and delivered to pharmacy practice faculty at 5 schools of pharmacy across the United States. Reliability was assessed by examining internal consistency, inter-rater reliability, and split-half reliability. Only responses which attained an inter-rater agreement>0.7 were included in the final version of the SJT. All responses were scored using a near-miss system, allowing higher scores for answers more closely aligned with the key, which was determined by the faculty who completed the SJT. Test fairness was reported using descriptive statistics. Thirty-nine faculty across the 5 participating institutions completed the SJT. The final version of the SJT included 105 responses, achieving an inter-rater agreement of>0.7 (inter-rater reliability of 0.98). Split-half reliability was 0.72. The average score was 85.7%, and no differences in performance were observed based on demographic characteristics. An SJT designed to assess empathy, integrity, and teamwork achieved reasonable levels of reliability among pharmacy practice faculty across the United States, and the results provided initial support for test validity and fairness. These results support a pilot to assess this SJT among students representing multiple institutions.

Decision-making

Empathy

Integrity

Situational judgment test

Teamwork

Details

Title: Subtitle: Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions
Creators: Kathryn J. Smith - Oklahoma City University
Brent N. Reed - University of North Carolina at Charlotte
Stephen Neely - Oklahoma City University
Michelle Z. Farland - Florida College
Stuart T. Haines - University of Mississippi
Jennifer D. Robinson - Washington State University Spokane
Resource Type: Journal article
Publication Details: American journal of pharmaceutical education, Vol.87(11), pp.100129-100129
DOI: 10.1016/j.ajpe.2023.100129
PMID: 37914464
NLM abbreviation: Am J Pharm Educ
ISSN: 0002-9459
eISSN: 1553-6467
Publisher: Elsevier Inc
Language: English
Date published: 11/01/2023
Academic Unit: Pharmacy Practice and Science
Record Identifier: 9984721127102771

Metrics

61 Record Views

1 Times Cited - Web of Science