Who r u?: On the (in)accuracy of incumbent-based estimates of range restriction in criterion-related and differential validity research

Philip L Roth; Huy Le; In-Sue Oh; Chad H Van Iddekinge; Steven B Robbins

doi:10.1037/apl0000193

Back

Who r u?: On the (in)accuracy of incumbent-based estimates of range restriction in criterion-related and differential validity research

Journal article

Open access

Peer reviewed

Who r u?: On the (in)accuracy of incumbent-based estimates of range restriction in criterion-related and differential validity research

Philip L Roth, Huy Le, In-Sue Oh, Chad H Van Iddekinge and Steven B Robbins

Journal of applied psychology, Vol.102(5), pp.802-828

05/01/2017

DOI: 10.1037/apl0000193

PMID: 28150987

Files and links (1)

url

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2902033View

Open Access

Abstract

Correcting validity estimates for selection procedures for range restriction typically involves comparing variance in predictor scores between all job applicants and applicants who were selected. However, some research on criterion-related and differential validity of cognitive ability tests has relied on range restriction corrections based on data from job incumbents. Unfortunately, there remains ambiguity concerning the accuracy of this incumbent-based approach vis-à-vis the applicant-based approach. To address this issue, we conducted several Monte Carlo simulations, as well as an analysis of college admissions data. Our first simulation study showed that incumbent-based range restriction corrections result in downwardly biased estimates of criterion-related validity, whereas applicant-based corrections were quite accurate. Our second set of simulations showed that incumbent-based range restriction corrections can produce evidence of differential validity when there is no differential validity in the population. In contrast, applicant-based corrections tended to accurately estimate population parameters and showed little, if any, evidence of differential validity when there is no differential validity in the population. Analysis of data for the ACT as a predictor of academic performance revealed similar patterns of bias for incumbent-based corrections in an academic setting. Overall, the present findings raise serious concerns regarding the use of incumbent-based range restriction corrections in lieu of applicant-based corrections. They also cast doubt on recent evidence for differential validity of predictors of job performance. (PsycINFO Database Record

Academic Performance

Aptitude Tests

Computer Simulation

Humans

Personnel Selection

Details

Title: Subtitle: Who r u?: On the (in)accuracy of incumbent-based estimates of range restriction in criterion-related and differential validity research
Creators: Philip L Roth - Clemson University
Huy Le - The University of Texas at San Antonio
In-Sue Oh - Temple University
Chad H Van Iddekinge - Florida State University
Steven B Robbins - Educational Testing Service
Resource Type: Journal article
Publication Details: Journal of applied psychology, Vol.102(5), pp.802-828
DOI: 10.1037/apl0000193
PMID: 28150987
NLM abbreviation: J Appl Psychol
ISSN: 0021-9010
eISSN: 1939-1854
Language: English
Date published: 05/01/2017
Academic Unit: Management and Entrepreneurship
Record Identifier: 9984380429002771

Metrics

19 Record Views

15 Times Cited - Web of Science