Modeling visual search behavior of breast radiologists using a deep convolution neural network

Suneeta Mall; Patrick C Brennan; Claudia Mello-Thoms

doi:10.1117/1.JMI.5.3.035502

Back

Modeling visual search behavior of breast radiologists using a deep convolution neural network

Journal article

Open access

Peer reviewed

Modeling visual search behavior of breast radiologists using a deep convolution neural network

Suneeta Mall, Patrick C Brennan and Claudia Mello-Thoms

Journal of medical imaging (Bellingham, Wash.), Vol.5(3), pp.035502-035502

07/2018

DOI: 10.1117/1.JMI.5.3.035502

PMCID: PMC6086967

PMID: 30128329

Files and links (1)

url

https://doi.org/10.1117/1.JMI.5.3.035502View

Published (Version of record) Open Access

Abstract

Visual search, the process of detecting and identifying objects using eye movements (saccades) and foveal vision, has been studied for identification of root causes of errors in the interpretation of mammograms. The aim of this study is to model visual search behavior of radiologists and their interpretation of mammograms using deep machine learning approaches. Our model is based on a deep convolutional neural network, a biologically inspired multilayer perceptron that simulates the visual cortex and is reinforced with transfer learning techniques. Eye-tracking data were obtained from eight radiologists (of varying experience levels in reading mammograms) reviewing 120 two-view digital mammography cases (59 cancers), and it has been used to train the model, which was pretrained with the ImageNet dataset for transfer learning. Areas of the mammogram that received direct (foveally fixated), indirect (peripherally fixated), or no (never fixated) visual attention were extracted from radiologists' visual search maps (obtained by a head mounted eye-tracking device). These areas along with the radiologists' assessment (including confidence in the assessment) of the presence of suspected malignancy were used to model: (1) radiologists' decision, (2) radiologists' confidence in such decisions, and (3) the attentional level (i.e., foveal, peripheral, or none) in an area of the mammogram. Our results indicate high accuracy and low misclassification in modeling such behaviors.

deep learning

mammography

visual search

eye tracking

breast cancer

machine learning

behavior modeling

Details

Title: Subtitle: Modeling visual search behavior of breast radiologists using a deep convolution neural network
Creators: Suneeta Mall - University of Sydney, Faculty of Health Sciences, Medical Image Optimisation and Perception Research Group (MIOPeG), Lidcombe, New South Wales, Australia
Patrick C Brennan - University of Sydney, Faculty of Health Sciences, Medical Image Optimisation and Perception Research Group (MIOPeG), Lidcombe, New South Wales, Australia
Claudia Mello-Thoms - University of Sydney, Faculty of Health Sciences, Medical Image Optimisation and Perception Research Group (MIOPeG), Lidcombe, New South Wales, Australia
Resource Type: Journal article
Publication Details: Journal of medical imaging (Bellingham, Wash.), Vol.5(3), pp.035502-035502
DOI: 10.1117/1.JMI.5.3.035502
PMID: 30128329
PMCID: PMC6086967
NLM abbreviation: J Med Imaging (Bellingham)
ISSN: 2329-4302
eISSN: 2329-4310
Publisher: United States
Language: English
Date published: 07/2018
Academic Unit: Roy J. Carver Department of Biomedical Engineering; Radiology
Record Identifier: 9984051557702771

Metrics

12 Record Views

25 Times Cited - Web of Science