Journal article
Estimating latent reader-performance variability using the Obuchowski-Rockette method
Proceedings of SPIE, the international society for optical engineering, Vol.10952, pp.109520F-109520F-6
02/2019
DOI: 10.1117/12.2513106
PMID: 32390679
Abstract
We describe how the Obuchowski-Rockette (OR) method of analysis for multi-reader diagnostic studies can be used to estimate the variability of latent reader-performance outcomes, such as the area under the ROC curve (AUC). For a specific reader the latent or true reader performance outcome can conceptually be thought of as the average of the estimates that would result if the reader were to read a very large number of case samples. We note that for the sample sizes used in typical diagnostic studies, the latent reader-performance outcome is equal to the observed outcome minus measurement error. An often-cited study that assesses the variability of various reader-performance outcomes, including the AUC, is the study by Craig Beam et. al., “Variability in the Interpretation of Screening Mammograms by US Radiologists,” published in 1996. However, a problem with this type of study is that the variability estimates include measurement error. Thus this approach overestimates latent reader variability and gives variability estimates that are dependent on case sample size. The proposed method overcomes this problem. We illustrate the proposed method for 29 radiologists in Jordan, with each reading 60 chest computed tomography (CT) scans. Using the OR method we estimate the middle 95% range for latent AUC values to be 0.07; i.e., we estimate that 95% of radiologists differ by less than 0.07 in their ability to successfully discriminate between a pair of diseased and nondiseased cases. In contrast, the estimate for the 95% range for the observed AUCs is 0.18. Thus we see how the conventional method of describing variability of reader performance estimates can greatly overstate the variability of the true abilities of the readers.
Details
- Title: Subtitle
- Estimating latent reader-performance variability using the Obuchowski-Rockette method
- Creators
- Stephen L Hillis - Departments of Radiology and Biostatistics, University of Iowa, Iowa City, IA, USABadera Al Mohammad - Department of Medical Imaging and Radiation Sciences, Faculty of Health Sciences, The University of Sydney, Lidcombe, NSW, AustraliaPatrick C Brennan - Department of Medical Imaging and Radiation Sciences, Faculty of Health Sciences, The University of Sydney, Lidcombe, NSW, Australia
- Resource Type
- Journal article
- Publication Details
- Proceedings of SPIE, the international society for optical engineering, Vol.10952, pp.109520F-109520F-6
- DOI
- 10.1117/12.2513106
- PMID
- 32390679
- NLM abbreviation
- Proc SPIE Int Soc Opt Eng
- ISSN
- 0277-786X
- eISSN
- 1996-756X
- Language
- English
- Date published
- 02/2019
- Academic Unit
- Radiology; Biostatistics
- Record Identifier
- 9984051539402771
Metrics
33 Record Views