Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification

Stephen L Hillis; Kevin S Berbaum

doi:10.1016/j.acra.2005.07.012

Back

Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification

Journal article

Peer reviewed

Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification

Stephen L Hillis and Kevin S Berbaum

Academic radiology, Vol.12(12), pp.1534-1541

12/2005

DOI: 10.1016/j.acra.2005.07.012

PMCID: PMC1550352

PMID: 16321742

Files and links (1)

url

http://doi.org/10.1016/j.acra.2005.07.012View

Open Access

Abstract

Two problems of the Dorfman-Berbaum-Metz (DBM) method for analyzing multireader receiver operating characteristic (ROC) studies are that it tends to be conservative and that it can produce AUC estimates outside the parameter space--ie, greater than one or less than zero. Recently it has been shown that the problem of AUC (or other accuracy) estimates outside the parameter space can be eliminated by using normalized pseudovalues, and it has been suggested that less data-based model simplification be used. Our purpose is to empirically investigate if these two modifications--normalized pseudovalues and less data-based model simplification--result in improved performance. We examine the performance of the DBM procedure using the two proposed modifications for discrete and continuous ratings in a null simulation study comparing modalities with respect to the ROC area. The simulation study includes 144 different combinations of reader and case sample sizes, normal/abnormal case sample ratios, and variance components. The ROC area is estimated using parametric and nonparametric estimation. The DBM procedure with both modifications performs better than either the original DBM procedure or the DBM procedure with only one of the modifications. For parametric estimation with discrete rating data, use of both modifications resulted in the mean type I error (0.043) closest to the nominal .05 level and the smallest range (0.050) and standard deviation (0.0108) across the 144 type I error rates. We recommend that normalized pseudovalues and less data-based model simplification be used with the DBM procedure.

Algorithms

Data Interpretation, Statistical

Sample Size

Radiography - methods

Computer Simulation

ROC Curve

Models, Statistical

Monte Carlo Method

Quality Assurance, Health Care - methods

Details

Title: Subtitle: Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification
Creators: Stephen L Hillis - Center for Research in the Implementation of Innovative Strategies in Practice (CRIISP), Iowa City VA Medical Center , 601 Highway 6 West, Iowa City, IA 52246-2208, USA. steve-hillis@uiowa.edu
Kevin S Berbaum
Resource Type: Journal article
Publication Details: Academic radiology, Vol.12(12), pp.1534-1541
DOI: 10.1016/j.acra.2005.07.012
PMID: 16321742
PMCID: PMC1550352
NLM abbreviation: Acad Radiol
ISSN: 1076-6332
eISSN: 1878-4046
Publisher: United States
Grant note: R01EB000863 / NIBIB NIH HHS R01 EB000863 / NIBIB NIH HHS
Language: English
Date published: 12/2005
Academic Unit: Radiology; Biostatistics
Record Identifier: 9984051715702771

Metrics

21 Record Views

65 Times Cited - Web of Science