Journal article
Reliability and Validity of Performance Evaluations of Pain Medicine Clinical Faculty by Residents and Fellows Using a Supervision Scale
Anesthesia and analgesia, Vol.131(3), pp.909-916
09/2020
DOI: 10.1213/ANE.0000000000004779
PMID: 32332292
Abstract
Annual and/or semiannual evaluations of pain medicine clinical faculty are mandatory for multiple organizations in the United States. We evaluated the validity and psychometric reliability of a modified version of de Oliveira Filho et al clinical supervision scale for this purpose. Six years of weekly evaluations of pain medicine clinical faculty by resident physicians and pain medicine fellows were studied. A 1-4 rating (4 = "Always") was assigned to each of 9 items (eg, "The faculty discussed with me the management of patients before starting a procedure or new therapy and accepted my suggestions, when appropriate"). Cronbach α of the 9 items equaled .975 (95% confidence interval [CI], 0.974-0.976). A G coefficient of 0.90 would be expected with 18 raters; the N = 12 six-month periods had mean 18.8 ± 5.9 (standard deviation [SD]) unique raters in each period (median = 20).Concurrent validity was shown by Kendall τb = 0.45 (P < .0001) pairwise by combination of ratee and rater between the average supervision score and the average score on a 21-item evaluation completed by fellows in pain medicine. Concurrent validity also was shown by τb = 0.36 (P = .0002) pairwise by combination of ratee and rater between the average pain medicine supervision score and the average operating room supervision score completed by anesthesiology residents.Average supervision scores differed markedly among the 113 raters (η = 0.485; CI, 0.447-0.490). Pairings of ratee and rater were nonrandom (Cramér V = 0.349; CI, 0.252-0.446).Mixed effects logistic regression was performed with rater leniency as covariates and the dependent variable being an average score equaling the maximum 4 vs <4. There were 3 of 13 ratees with significantly more averages <4 than the other ratees, based on P < .01 criterion; that is, their supervision was reliably rated as below average. There were 3 of 13 different ratees who provided supervision reliably rated as above average.Raters did not report higher supervision scores when they had the opportunity to perform more interventional pain procedures. Evaluations of pain medicine clinical faculty are required. As found when used for evaluating operating room anesthesiologists, a supervision scale has excellent internal consistency, achievable reliability using 1-year periods of data, concurrent validity with other ratings, and the ability to differentiate among ratees. However, to be reliable, routinely collected supervision scores must be adjusted for rater leniency.
Details
- Title: Subtitle
- Reliability and Validity of Performance Evaluations of Pain Medicine Clinical Faculty by Residents and Fellows Using a Supervision Scale
- Creators
- Franklin Dexter - From the Division of Management Consulting, Department of Anesthesia, University of Iowa, Iowa City, IowaKatherine Hadlandsmyth - Department of Anesthesia, University of Iowa, Iowa City, IowaAmy C S Pearson - Department of Anesthesia, University of Iowa, Iowa City, IowaBradley J Hindman - Department of Anesthesia, University of Iowa, Iowa City, Iowa
- Resource Type
- Journal article
- Publication Details
- Anesthesia and analgesia, Vol.131(3), pp.909-916
- Publisher
- United States
- DOI
- 10.1213/ANE.0000000000004779
- PMID
- 32332292
- ISSN
- 0003-2999
- eISSN
- 1526-7598
- Language
- English
- Electronic publication date
- 04/22/2020
- Date published
- 09/2020
- Academic Unit
- Psychiatry; Health Management and Policy; Anesthesia
- Record Identifier
- 9983806383502771
Metrics
19 Record Views