Journal article
Roe and Metz identical-test simulation model for validating multi-reader methods of analysis for comparing different radiologic imaging modalities
Journal of medical imaging (Bellingham, Wash.), Vol.10(S1), S11916
07/01/2023
DOI: 10.1117/1.JMI.10.S1.S11916
PMCID: PMC10321333
PMID: 37416057
Abstract
Purpose
The most frequently used model for simulating multi-reader multi-case (MRMC) data that emulate confidence-of-disease ratings from diagnostic imaging studies has been the Roe and Metz model, proposed by Roe and Metz in 1997 and later generalized by Hillis (2012), Abbey et al. (2013), and Gallas and Hillis (2014). These models have been used for evaluating MRMC analysis and sample size methods. The models suggested in these papers for assessing type I error have been null models, where the expected area under the receiver-operating-characteristic curve across readers is the same for each test. However, for these null models, there are other differences that would not exist if the two tests were identical. None of the papers mentioned above discuss how to formulate a null model that is also an identical-test model, where the two tests are identical in all respects. The purpose of this paper is to show how to formulate a Roe and Metz identical-test model and to show its usefulness for validating the error covariance constraints employed by the Obuchowski-Rockette (1995) method.
Approach
For a given Roe-and-Metz model, the corresponding Roe-and-Metz identical-test model is derived by modifying the Roe and Metz null model under the assumption that the two tests are identical.
Results
The importance of the Obuchowski-Rockette model constraints for avoiding negative variance estimates is established using data simulated from the Roe and Metz identical-test model. It is also shown that negative variance estimates can occur at nontrivial rates when the two tests are not identical but somewhat “close” to being identical.
Conclusions
The findings of this paper are important because it has recently been shown (Hillis, 2022) that the commonly used MRMC method proposed by Gallas (2006) and Gallas et al. (2009) uses the same test statistic as the unconstrained Obuchowski-Rockette method.
Details
- Title: Subtitle
- Roe and Metz identical-test simulation model for validating multi-reader methods of analysis for comparing different radiologic imaging modalities
- Creators
- Stephen L. Hillis - University of Iowa
- Resource Type
- Journal article
- Publication Details
- Journal of medical imaging (Bellingham, Wash.), Vol.10(S1), S11916
- DOI
- 10.1117/1.JMI.10.S1.S11916
- PMID
- 37416057
- PMCID
- PMC10321333
- NLM abbreviation
- J Med Imaging (Bellingham)
- ISSN
- 2329-4302
- eISSN
- 2329-4310
- Publisher
- SPIE
- Number of pages
- 28
- Grant note
- National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health: R01EB025174 National Institute of Biomedical Imaging and Bioengineering: R01EB025174
This research was supported by the National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health (Grant No. R01EB025174). Some of the information presented in this paper was presented in a prior SPIE proceedings paper by the author.24 I thank two reviewers and the editor for their very helpful comments and suggestions. This content is solely the responsibility of the author and does not necessarily represent the official views of the National Institutes of Health.
- Language
- English
- Date published
- 07/01/2023
- Description audience
- Academic
- Academic Unit
- Radiology; Biostatistics
- Record Identifier
- 9985091805202771
Metrics
3 Record Views