Dissertation
A comparative study of IRT models for rater effects and double scoring
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Autumn 2019
DOI: 10.17077/etd.005229
Abstract
In dealing with rater effects, double scoring is a popular method to control the quality of ratings for tests including constructed-response (CR) type of items. Treating individual multiple ratings as independent violates the local independence assumption in item response theory (IRT). The typical way to fit standard IRT models to multiple ratings is to use the linear combination of multiple ratings as item scores, such as sum or average scores. However, these summed or averaged score approaches have limitations because it requires the adjustment of original item score categories and still contains rater effects in item scores. The purpose of this dissertation is to assess the effectiveness of using double ratings over single ratings in standard IRT models when rater effects are present, and to compare the performance of standard and newer IRT models for rater effects and multiple ratings, known to correct rater effects from parameter estimation and preserve the original item score categories.
Two simulation studies examined the accuracy of IRT models. As such, the number of ratings and IRT models were considered as main factors in the simulation study. The number of ratings includes single and double ratings. Two IRT models entail the generalized partial credit model (GPCM) and hierarchical rater model (HRM), each representing a standard IRT model and the IRT model for multiple ratings and rater effects. The HRM was used to generate ratings with rater effects. Then the GPCM and HRM were fitted to ratings. All the ratings were generated with the combination of other study factors, including sample size, test length, rater effects, and number of score categories. Results were compared and interpreted relative to baseline conditions, where ratings were generated with no rater effects.
The main findings of this dissertation were as follows: (1) using single ratings as item scores in rater effect conditions reduced the accuracy of proficiency estimation in the GPCM; (2) double scoring methods relieved the impact of rater effects on proficiency estimation and improved accuracy in the GPCM; (3) for double ratings, the HRM showed better performance than the GPCM using summed item scores; (4) as more items and larger number of score categories were used, accuracy of proficiency estimation improved, in general.
Details
- Title: Subtitle
- A comparative study of IRT models for rater effects and double scoring
- Creators
- Yoon Ah Song
- Contributors
- Brandon C LeBeau (Advisor)Won-Chan Lee (Advisor)Jonathan Templin (Committee Member)Ariel Aloe (Committee Member)Sanvesh Srivastava (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Psychological and Quantitative Foundations
- Date degree season
- Autumn 2019
- DOI
- 10.17077/etd.005229
- Publisher
- University of Iowa
- Number of pages
- xvi, 150 pages
- Copyright
- Copyright 2019 Yoon Ah Song
- Comment
- This thesis has been optimized for improved web viewing. If you require the original version, contact the University Archives at the University of Iowa: https://www.lib.uiowa.edu/sc/contact/
- Language
- English
- Description illustrations
- illustrations
- Description bibliographic
- Includes bibliographical references (pages 125-136).
- Public Abstract (ETD)
Item Response Theory (IRT) is a statistical approach which measures examinees proficiency and evaluates test items based on examinees' responses. When tests consist of multiple choice items, scores are directly determined by responses and no raters are involved. However, when tests require examinees to construct answers to test items (constructed-response, CR) and scores are assigned by a rater's evaluation, it is possible that incorrect scores might be assigned. To control such rater effects, double scoring has been a popular procedure, in which responses are scored twice by different raters. The purpose of this dissertation study is to (1) evaluate whether double scoring is an effective method to improve scoring accuracy with CR items that use a standard IRT model not considering rater effects. Another purpose is to (2) evaluate the extent to which a newer IRT model improves scoring accuracy compared to the standard IRT model. The results show that double scoring methods relieved the negative impact of rater effects and improved scoring accuracy. The new IRT model showed better performance than the standard IRT model. Although CR items comprise a significant portion of educational assessments in most testing programs, the use of IRT models have been limited to a few standard IRT models that are often not compatible with a double scoring practice. This study showed the possibility to better control rating qualities for tests which contain CR items by providing information on the performance of different IRT models.
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9983779900002771
Metrics
105 File views/ downloads
168 Record Views