Journal article
Application of Threshold-bias Independent Analysis to Eye-tracking and FROC Data
Academic radiology, Vol.19(12), pp.1474-1483
12/2012
DOI: 10.1016/j.acra.2012.09.002
PMCID: PMC3489965
PMID: 23040503
Abstract
Studies of medical image interpretation have focused on either assessing radiologists' performance using, for example, the receiver operating characteristic (ROC) paradigm, or assessing the interpretive process by analyzing their eye-tracking (ET) data. Analysis of ET data has not benefited from threshold-bias independent figures of merit (FOMs) analogous to the area under the receiver operating characteristic (ROC) curve. The aim was to demonstrate the feasibility of such FOMs and to measure the agreement between FOMs derived from free-response ROC (FROC) and ET data.
Eight expert breast radiologists interpreted a case set of 120 two-view mammograms while eye-position data and FROC data were continuously collected during the interpretation interval. Regions that attract prolonged (>800 ms) visual attention were considered to be virtual marks, and ratings based on the dwell and approach-rate (inverse of time-to-hit) were assigned to them. The virtual ratings were used to define threshold-bias independent FOMs in a manner analogous to the area under the trapezoidal alternative FROC (AFROC) curve (0 = worst, 1 = best). Agreement at the case level (0.5 = chance, 1 = perfect) was measured using the jackknife and 95% confidence intervals (CI) for the FOMs and agreement were estimated using the bootstrap.
The AFROC mark-ratings' FOM was largest at 0.734 (CI 0.65–0.81) followed by the dwell at 0.460 (0.34–0.59) and then by the approach-rate FOM 0.336 (0.25–0.46). The differences between the FROC mark-ratings' FOM and the perceptual FOMs were significant (P < .05). All pairwise agreements were significantly better then chance: ratings vs. dwell 0.707 (0.63–0.88), dwell vs. approach-rate 0.703 (0.60–0.79) and rating vs. approach-rate 0.606 (0.53–0.68). The ratings vs. approach-rate agreement was significantly smaller than the dwell vs. approach-rate agreement (P = .008).
Leveraging current methods developed for analyzing observer performance data could complement current ways of analyzing ET data and lead to new insights.
Details
- Title: Subtitle
- Application of Threshold-bias Independent Analysis to Eye-tracking and FROC Data
- Creators
- Dev P Chakraborty - Department of Radiology, University of Pittsburgh, Presbyterian South Tower, Room 4771, 200 Lothrop Street, Pittsburgh, PA 15213Hong-Jun Yoon - Department of Radiology, University of Pittsburgh, Presbyterian South Tower, Room 4771, 200 Lothrop Street, Pittsburgh, PA 15213Claudia Mello-Thoms - Department of Biomedical Informatics and Department of Radiology, University of Pittsburgh School of Medicine, Pittsburgh, PA
- Resource Type
- Journal article
- Publication Details
- Academic radiology, Vol.19(12), pp.1474-1483
- DOI
- 10.1016/j.acra.2012.09.002
- PMID
- 23040503
- PMCID
- PMC3489965
- NLM abbreviation
- Acad Radiol
- ISSN
- 1076-6332
- eISSN
- 1878-4046
- Publisher
- Elsevier Inc
- Grant note
- R01 EB008688 / NIH/NIBIB K01 HS018365 / AHRQ
- Language
- English
- Date published
- 12/2012
- Academic Unit
- Roy J. Carver Department of Biomedical Engineering; Radiology
- Record Identifier
- 9984051523302771
Metrics
26 Record Views