Dissertation
Investigating hybrid test designs in passage-based adaptive tests
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Summer 2020
DOI: 10.17077/etd.005590
Abstract
Computerized adaptive testing (CAT) selects each item to match with an examinee’s ability level during the test. In contrast, multistage adaptive testing (MST) administers a group of items instead of each individual item adaptively. A hybrid design combines CAT and MST borrowing strengths from both testing modes. However, existing research studies indicate that there is lack of knowledge on hybrid designs in passage-based adaptive testing, including situations where misrouting occurs. Misrouting refers to when examinees get routed to paths that don’t match with their true ability levels. Therefore, the purpose of this study is to evaluate the proposed hybrid designs (HMCAT designs) with respect to ability estimation accuracy under different MST configurations and their performance when misrouting occurs.
The HMCAT designs under passage-based adaptive testing control test adaptability at both passage/stimulus level and item level. Four proposed HMCAT designs differ by implementing the item level CAT at different stages of the MST. Specifically, among all four designs, the two most extreme designs are the one implementing CAT across all stages (CC) and the one not implementing CAT across any stage (PP). The two hybrid designs include implementing CAT only in the routing stage (CP) and implementing CAT only in the last stage (PC). The hybrid designs’ performance compared to that of the CC design and the PP design when interacting with different MST configurations and when misrouting occurs are the two main research questions of the study.
The three-phase simulation analysis indicates that (1) the PC design achieves more accurate final ability estimation results under the three-stage and the four-stage MST configurations; (2) although the CP design fails to perform effectively when misrouting occurs under the three-stage MST configuration, it is still able to achieve accurate final ability estimation results under both two-stage MST configurations; (3) the CC design and the PP design achieve similar and the most accurate final ability estimation results under all four MST configurations; (4) the PC design reaches similar accuracy as the CC design and the PP design under the three-stage and the four-stage MST configurations; (5) the CP design reaches a similar accuracy level as the CC design and the PP design under both two-stage MST configurations; (6) the CP design is affected the most when misrouting occurs in the three-stage MST configuration, whereas the other three HMCAT designs are not affected.
In general, findings from simulation analyses suggest that the HMCAT designs are effective enough to use in passage/stimulus-based adaptive tests with appropriate MST configurations. They also contain advantages such as greater flexibility with respect to content balancing, test security and control on item administration. However, it is important to understand that the performance of HMCAT designs can heavily depend on the MST configuration, the item pool characteristics, the test design and components in the test administration algorithm, such as content balancing strategies. Results from the current study provide implications for practitioners on how to decide on an appropriate HMCAT design under real testing contexts and how to evaluate and maintain the selected HMCAT design to effectively implement the design in practice.
Details
- Title: Subtitle
- Investigating hybrid test designs in passage-based adaptive tests
- Creators
- Ye Cheryl Ma
- Contributors
- Stephen B Dunbar (Advisor)Deborah J Harris (Advisor)Catherine J Welch (Committee Member)Terry A Ackerman (Committee Member)Robert D Ankenmann (Committee Member)Sanvesh Srivastava (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Psychological and Quantitative Foundations
- Date degree season
- Summer 2020
- DOI
- 10.17077/etd.005590
- Publisher
- University of Iowa
- Number of pages
- xviii, 204 pages
- Copyright
- Copyright 2020 Ye Cheryl Ma
- Language
- English
- Description illustrations
- color illustrations
- Description bibliographic
- Includes bibliographical references (page 188-197).
- Public Abstract (ETD)
Computerized adaptive testing (CAT) and multistage adaptive testing (MST) are the two most popular adaptive testing modes. A hybrid design combines CAT and MST into one design borrowing strengths from both modes. Because there is lack of knowledge on hybrid designs in passage-based adaptive testing and their performance when misrouting (misrouting is when examinees get routed to paths that don’t match with their true ability levels) occurs, the purpose of this study is to evaluate the proposed hybrid designs (HMCAT designs) with respect to ability estimation accuracy under different MST configurations and their performance when misrouting occurs.
This study evaluated the performance of the four proposed HMCAT designs under four different MST configurations. The evaluation was conducted via a three-phrase simulation analysis. The HMCAT designs’ performance when misrouting occurs under a three-stage MST configuration was also investigated.
Findings from the simulation analyses indicate that the HMCAT designs can be effective in terms of resulting in accurate ability estimation under certain MST configurations. Three of the HMCAT designs were able to effectively estimate final ability even when misrouting occurs. Factors including item pool characteristics, test designs, and components in the test administration algorithms are important to consider. Results from this study should assist practitioners on how to decide on and maintain an appropriate HMCAT design to implement in practice.
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9983987998902771
Metrics
118 File views/ downloads
128 Record Views