Logo image
Back
Measuring test performance with signal detection theory techniques
Book chapter

Measuring test performance with signal detection theory techniques

Teresa A Treat and Richard J Viken
APA handbook of research methods in psychology, Vol 1: Foundations, planning, measures, and psychometrics, pp.723-744
American Psychological Association
2012
DOI: 10.1037/13619-038

View Online

Abstract

The development and evaluation of assessment and prediction strategies designed to distinguish two mutually exclusive states are central enterprises in psychological science. For example, we might want to assess diagnostic status (present or absent) or child maltreatment (present or absent). Alternatively, we might be interested in predicting whether violence is likely or whether treatment relapse will occur. Once classic psychometric methods have been used to develop one or more assessment devices, the predictive or criterion validity of the measurement strategies must be evaluated (see Clark & Watson, 1995; Smith, 2005). Widely used indexes of test performance include Sensitivity (the proportion of positive cases correctly classified as positive), Specificity (the proportion of negative cases correctly classified as negative), Positive Predictive Power (the proportion of cases classified as positive who actually are positive), and Negative Predictive Power (the proportion of cases classified as negative who actually are negative). However, these indexes vary widely as a function of cutoff scores, the base rates (BRs) of the phenomenon of interest, and the costs and benefits associated with a particular assessment or prediction context, as we will see. As a result, researchers increasingly are relying on the methods of signal detection theory, particularly receiver operating characteristic (ROC) analysis and utility-based decision theory approaches. ROC methods can be used to quantify and compare the discriminative power of measurement devices independently of cutoff scores, BRs, and costs and benefits. Decision theory methods then can be used to determine optimal cutoff scores for particular contexts, given specification of the BRs and the values placed on different kinds of correct and incorrect decisions. After presenting background on the role of BRs in assessment and prediction as well as traditional accuracy indexes, we provide an overview of the use of ROC and decision-theory approaches for examination and enhancement of decision making in psychology. We close with recommendations regarding the reporting of the development of new measures, especially with regard to optimal cutoff values for a range of BRs and several common decision goals. (PsycINFO Database Record (c) 2014 APA, all rights reserved)
 Research Methods & Experimental Design

Details

Metrics

23 Record Views
Logo image