Journal article
Reliability of ranking anesthesiologists and nurse anesthetists using leniency-adjusted clinical supervision and work habits scores
Journal of Clinical Anesthesia, Vol.61, pp.109639-109639
05/2020
DOI: 10.1016/j.jclinane.2019.109639
PMID: 31735571
Abstract
Evaluation of faculty anesthesiologists' clinical supervision is psychometrically reliable. Supervision scores often are used for ongoing professional practice and teaching evaluations. We evaluated whether anesthesiologists' clinical supervision rank could be determined reliably using 6- vs. 12-month data collection intervals and, for each, determined specificity (quartiles vs. halves). To serve as a comparator/control group, we analyzed anesthesiologists' evaluations of the work habits of nurse anesthetists. Residents evaluated 73 anesthesiologists and anesthesiologists evaluated 72 nurse anesthetists with whom they worked in operating rooms for ≥60 min, daily. Scores were adjusted for rater leniency. The reliability of ranking in halves and quartiles was determined and comparisons made. Clinical supervision and work habit scores, respectively. Using 1 year of data, 5% of anesthesiologists had scores that were reliably in the lowest quartile and 21% had scores that were reliably in the lower half. No (0%) anesthesiologists were reliably in the highest quartile and 19% were reliably in the upper half. Corresponding percentages for nurse anesthetists' work habits scores were 3%, 10%, 0%, and 1%. The primary factor limiting the reliability of ranking in quartiles was the number of operating room days worked. Ranks based on raw (unadjusted) scores falsely misclassified 60% of anesthesiologists and 88% of nurse anesthetists. The percentages of anesthesiologists who could be ranked reliably was less when using a shorter evaluation period (6 months). Using mixed effects regression to control for rater leniency, anesthesiologists' ranks based on supervision scores can be determined reliably for halves and quartiles, but fewer than half the anesthesiologists will be reliably ranked. The same ranking principles can be applied for evaluation of nurse anesthetists' work habits. Even when very high G-scores of 0.90 are obtained, ranking individuals into smaller groups (e.g., deciles) or using raw (observed) ranks is unreliable. •Can anesthesiologists' clinical supervision ranks be estimated reliably using 6-month and 12-month data collection intervals?•No, using 1 year of data, 5% of anesthesiologists had scores reliably in the lowest quartile and 0% reliably in the highest quartile.•Ranks based on raw (unadjusted) scores falsely misclassified 60% of anesthesiologists.•Even with very high G-scores (0.90), ranking individuals into smaller groups (e.g., deciles) or using raw (observed) ranks is unreliable.
Details
- Title: Subtitle
- Reliability of ranking anesthesiologists and nurse anesthetists using leniency-adjusted clinical supervision and work habits scores
- Creators
- Franklin Dexter - Department of Anesthesia, University of Iowa, United States of AmericaEmine O Bayman - Department of Biostatistics, University of Iowa, United States of AmericaCynthia A Wong - Department of Anesthesia, University of Iowa, United States of AmericaBradley J Hindman - Department of Anesthesia, University of Iowa, United States of America
- Resource Type
- Journal article
- Publication Details
- Journal of Clinical Anesthesia, Vol.61, pp.109639-109639
- Publisher
- Elsevier Inc
- DOI
- 10.1016/j.jclinane.2019.109639
- PMID
- 31735571
- ISSN
- 0952-8180
- eISSN
- 1873-4529
- Language
- English
- Date published
- 05/2020
- Academic Unit
- Health Management and Policy; Biostatistics; Anesthesia
- Record Identifier
- 9983806384802771
Metrics
16 Record Views