Journal article
Spatial localization accuracy of radiologists in free-response studies: Inferring perceptual FROC curves from mark-rating data
Academic radiology, Vol.14(1), pp.4-18
01/2007
DOI: 10.1016/j.acra.2006.10.015
PMCID: PMC1829298
PMID: 17178361
Abstract
Free-response data consist of a set of mark-ratings pairs. Before analysis, the data are classified or "scored" into lesion and non-lesion localizations. The scoring is done by choosing an acceptance-radius and classifying marks within the acceptance-radius of lesion centers as lesion localizations, and all other marks are classified as non-lesion localizations. The scored data are plotted as a free-response receiver operating characteristic (FROC) curve, essentially a plot of appropriately normalized numbers of lesion localizations vs. non-lesion localizations. Scored FROC curves are frequently used to compare imaging systems and computer-aided detection (CAD) algorithms. However, the choice of acceptance-radius is arbitrary. This makes it difficult to compare curves from different studies and to estimate true performance.
To resolve this issue the concept of two types of marks is introduced: perceptual hits and perceptual misses. A perceptual hit is a mark made in response to the observer seeing the lesion. A perceptual miss is a mark made in response to the observer seeing a (lesion-like) non-lesion. A method of estimating the most probable numbers of perceptual hits and misses is described. This allows one to plot a perceptual FROC operating point and by extension a perceptual FROC curve. Unlike a scored FROC operating point, a perceptual point is independent of the choice of acceptance-radius. The method does not allow one to identify individual marks as perceptual hits or misses-only the most probable numbers. It is based on a three-parameter statistical model of the spatial distributions of perceptual hits and misses relative to lesion centers.
The method has been applied to an observer dataset in which mammographers and residents with different levels of experience were asked to locate lesions in mammograms. The perceptual operating points suggest superior performance for the mammographers and equivalent performance for residents in the first and second mammography rotations. These results and the model validation are preliminary as they are based on a small dataset.
The significance of this study is showing that it is possible to probabilistically determine if a mark resulted from seeing a lesion or a non-lesion. Using the method developed in this study one could perform acceptance-radius independent estimation of observer performance.
Details
- Title: Subtitle
- Spatial localization accuracy of radiologists in free-response studies: Inferring perceptual FROC curves from mark-rating data
- Creators
- Dev Chakraborty - Department of Radiology, University of Pittsburgh, 3520 5th Avenue, Suite 300, Pittsburgh, PA 15261, USA. dpc10@pitt.eduHong-Jun YoonClaudia Mello-Thoms
- Resource Type
- Journal article
- Publication Details
- Academic radiology, Vol.14(1), pp.4-18
- DOI
- 10.1016/j.acra.2006.10.015
- PMID
- 17178361
- PMCID
- PMC1829298
- NLM abbreviation
- Acad Radiol
- ISSN
- 1076-6332
- eISSN
- 1878-4046
- Publisher
- United States
- Grant note
- R01 EB005243 / NIBIB NIH HHS R01EB005243 / NIBIB NIH HHS
- Language
- English
- Date published
- 01/2007
- Academic Unit
- Roy J. Carver Department of Biomedical Engineering; Radiology
- Record Identifier
- 9984051764602771
Metrics
35 Record Views