Task-Based Assessment for Neural Networks: Evaluating Undersampled MRI Reconstructions based on Human Observer Signal Detection

Joshua D Herman; Rachel E Roca; Alexandra G O'Neill; Marcus L Wong; Sajan G Lingala; Angel R Pineda

doi:10.48550/arXiv.2210.12161

Back

Task-Based Assessment for Neural Networks: Evaluating Undersampled MRI Reconstructions based on Human Observer Signal Detection

Preprint

Open access

Task-Based Assessment for Neural Networks: Evaluating Undersampled MRI Reconstructions based on Human Observer Signal Detection

Joshua D Herman, Rachel E Roca, Alexandra G O'Neill, Marcus L Wong, Sajan G Lingala and Angel R Pineda

ArXiv.org

10/21/2022

DOI: 10.48550/arXiv.2210.12161

Files and links (1)

url

https://doi.org/10.48550/arXiv.2210.12161View

Preprint (Author's original)This preprint has not been evaluated by subject experts through peer review. Preprints may undergo extensive changes and/or become peer-reviewed journal articles. Open Access

Abstract

Recent research has explored using neural networks to reconstruct undersampled magnetic resonance imaging (MRI) data. Because of the complexity of the artifacts in the reconstructed images, there is a need to develop task-based approaches of image quality. Common metrics for evaluating image quality like the normalized root mean squared error (NRMSE) and structural similarity (SSIM) are global metrics which average out impact of subtle features in the images. Using measures of image quality which incorporate a subtle signal for a specific task allow for image quality assessment which locally evaluates the effect of undersampling on a signal. We used a U-Net to reconstruct under-sampled images with 2x, 3x, 4x and 5x fold 1-D undersampling rates. Cross validation was performed for a 500 and a 4000 image training set with both structural similarity (SSIM) and mean squared error (MSE) losses. A two alternative forced choice (2-AFC) observer study was carried out for detecting a subtle signal (small blurred disk) from images with the 4000 image training set. We found that for both loss functions and training set sizes, the human observer performance on the 2-AFC studies led to a choice of a 2x undersampling but the SSIM and NRMSE led to a choice of a 3x undersampling. For this task, SSIM and NRMSE led to an overestimate of the achievable undersampling using a U-Net before a steep loss of image quality when compared to the performance of human observers in the detection of a subtle lesion.

Details

Title: Subtitle: Task-Based Assessment for Neural Networks: Evaluating Undersampled MRI Reconstructions based on Human Observer Signal Detection
Creators: Joshua D Herman - Mathematics Department, Manhattan College, NY
Rachel E Roca - Mathematics Department, Manhattan College, NY
Alexandra G O'Neill - Manhattan University
Marcus L Wong - Mathematics Department, Manhattan College, NY
Sajan G Lingala - University of Iowa
Angel R Pineda - Manhattan University
Resource Type: Preprint
Publication Details: ArXiv.org
DOI: 10.48550/arXiv.2210.12161
ISSN: 2331-8422
Language: English
Date posted: 10/21/2022
Academic Unit: Roy J. Carver Department of Biomedical Engineering; Radiology
Record Identifier: 9984306743102771

Metrics

105 Record Views