Dissertation
Comparison of some detection methods on unidimensional IRT calibration
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Autumn 2019
DOI: 10.17077/etd.005181
Abstract
Nowadays it is not uncommon that tests, especially high-stakes assessments, are administered with time constraints. When a test is constructed to assess examinees’ abilities in academic knowledge, but the imposed time limits affect examinees’ test performance, speededness effects become a concern. Under such circumstances, inaccurate psychometric results and inferences might be drawn if unidimensional item response theory (IRT) models are applied in testing practice.
Speededness detection methods were proposed to identify speeded responses/examinees. Thus, the purpose of the study was to comprehensively investigate how the performance of various detection methods combined with various calibration treatments compared in reducing speededness effects under the 2PL and 3PL IRT models with: (1) simulated test data under various speededness conditions, and (2) real test data.
Both simulated and real data analyses were conducted in this study. Two simulation studies were conducted. For the first simulation study, two main factors were investigated: (1) degree of speededness (three levels: None, 10%, and 25%), and (2) IRT calibration model (two models: 2PL, and 3PL). The performance of various combinations of detection methods and calibration treatments were evaluated by assessing Pearson correlation, item parameter recovery, and model-data fit statistics. Data generated in the second simulation study were based on the estimated person and item parameter values obtained from IRT model calibration of the real data used in this study. Thus, the second simulation study served as a link between the pure simulation study and the real data study, because such a generation process enabled the simulated dataset to carry some characteristics of the real data, while true parameter values were known. The real data came from a large pool of a high-stakes standardized assessment items.
In the current study, it was found that treating the identified speeded responses as “not-presented” could always lead to more accurate psychometric results compared to the other calibration treatments across various speededness levels under both the 2PL and 3PL IRT models. When the speededness level was large, “removing speeded examinees” could usually yield comparable results compared to “not-presented” treatments across different detection methods, and is a feasible and easily manipulated option in practice. In addition, it was found that detection methods using the item response time (RT) distribution as a speededness indicator (i.e., the INSPECT and VITP methods in the current study) generally showed better performance than the other detection methods in dealing with speededness effects. Moreover, in this study, it was found that the inclusion of the c-parameter could deal with rapid guessing strategy well. Thus, when the speededness level was not large, and mainly caused by rapid guessing behavior, “no treatment” under the 3PL IRT model yielded accurate psychometric results.
The findings of the current study provide several feasible options for practitioners when speededness is a concern and unidimentional IRT models are used in the calibration or scoring process. It is hoped that this study will inspire researchers and practitioners to develop new detection methods, or ways of dealing with speededness effects under unidimensional IRT models.
Details
- Title: Subtitle
- Comparison of some detection methods on unidimensional IRT calibration
- Creators
- Lu Wang
- Contributors
- Robert Ankenmann (Advisor)Catherine Welch (Committee Member)Deborah Harris (Committee Member)Stephen Dunbar (Committee Member)Kate Cowles (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Psychological and Quantitative Foundations
- Date degree season
- Autumn 2019
- DOI
- 10.17077/etd.005181
- Publisher
- University of Iowa
- Number of pages
- xiii, 147 pages
- Copyright
- Copyright 2019 Lu Wang
- Language
- English
- Description illustrations
- color illustrations
- Description bibliographic
- Includes bibliographical references (pages 90-95).
- Public Abstract (ETD)
Currently, unidimensional item response theory (UIRT) models are widely employed in calibration and scoring processes in large-scale tests. However, successful applications of UIRT models rely heavily on whether the unidimensional IRT assumptions are satisfied. When a test is intended to assess examinees’ ability levels, but time constraints are imposed, examinees’ test performance is affected not only by their ability levels, but also by how quickly they answer the test questions. That is, a speed component plays a nonnegligible role in affecting examinees’ test performance. Under such circumstances, using UIRT models could lead to a violation of both unidimensionality and local independence assumptions of the UIRT models (Hambleton & Swaminathan, 1985; Yen, 1993).
To avoid speededness effects on the application of UIRT models, several speededness detection methods were proposed to identify either speeded responses or examinees. This study compared the performance of various speededness detection methods, combined with various calibration treatments in reducing speededness effects under both 2PL and 3PL IRT models. The study found that item parameter estimates were more affected than person parameter estimates by speededness effects. In addition, treating the identified speeded responses as “not-presented” in calibration provided more accurate parameter estimates compared to using the other calibration treatments across various speededness levels under both the 2PL and 3PL IRT models. Moreover, the detection methods that were based on response time distribution to identify speeded responses showed better performance than the other detection methods in dealing with speededness effects.
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9983779398002771
Metrics
30 File views/ downloads
127 Record Views