Conference proceeding
Estimating maximal measurable performance for automated decision systems from the characteristics of the reference standard. application to diabetic retinopathy screening
2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vol.2014, pp.154-157
08/2014
DOI: 10.1109/EMBC.2014.6943552
PMID: 25569920
Abstract
We investigate the maximal performance that can be measured for automated binary decision systems in terms of area under the ROC curve (AUC), against a reference standard provided by human readers. The goal is to determine the required characteristics of the reference standard to assess and compare automated decision systems with a given degree of confidence, or, to determine what degree of confidence can be obtained given the characteristics of the reference standard. We modeled the expected value of the AUC that can be measured for a perfect decision system, given a reference standard provided either by a single human reader or by multiple human readers (consensus, majority vote). The proposed model was applied to diabetic retinopathy screening in a dataset of 874 eye fundus examinations graded by three readers. The expected value of the AUC for a perfect decision system was estimated at 0.956 against a single human reader, and 0.990 against a `majority wins' vote of three human readers. The Iowa detection program has reached the maximal performance measurable by a single human reader (0.929, CI: [0.897-0.962]) and is close to the maximal performance measurable by a `majority wins' vote (0.955, CI: [0.939-0.972]).
Details
- Title: Subtitle
- Estimating maximal measurable performance for automated decision systems from the characteristics of the reference standard. application to diabetic retinopathy screening
- Creators
- Gwenole Quellec - SFR ScInBioS, Inserm, Brest, FranceMichael D Abramoff - Dept. of Ophthalmology & Visual Sci., Univ. of Iowa, Iowa City, IA, USA
- Resource Type
- Conference proceeding
- Publication Details
- 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vol.2014, pp.154-157
- Publisher
- IEEE
- DOI
- 10.1109/EMBC.2014.6943552
- PMID
- 25569920
- ISSN
- 1094-687X
- eISSN
- 1558-4615
- Language
- English
- Date published
- 08/2014
- Academic Unit
- Roy J. Carver Department of Biomedical Engineering; Electrical and Computer Engineering; Ophthalmology and Visual Sciences
- Record Identifier
- 9983806392102771
Metrics
29 Record Views